Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:57:55.720145 18380 test_util.cc:268] Using random seed: -293352509
I20240502 01:57:55.725973 18380 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:57:55.726019 18380 ts_itest-base.cc:113] --------------
I20240502 01:57:55.726034 18380 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:57:55.726050 18380 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:57:55.726063 18380 ts_itest-base.cc:116] --------------
2024-05-02T01:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:57:55Z Disabled control of system clock
I20240502 01:57:55.734459 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:38681
--webserver_interface=127.17.243.62
--webserver_port=0
--builtin_ntp_servers=127.17.243.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.243.62:38681
--enable_range_replica_placement=false with env {}
W20240502 01:57:55.832705 18392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:55.832901 18392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:55.832942 18392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:55.836019 18392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:57:55.836072 18392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:55.836091 18392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:57:55.836120 18392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:57:55.839594 18392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.243.62:38681
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:38681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.243.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.18392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:55.840240 18392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:55.841149 18392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:55.847802 18399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:55.848192 18401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:55.848582 18398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:55.849028 18392 server_base.cc:1034] running on GCE node
I20240502 01:57:55.849431 18392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:55.850221 18392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:55.851398 18392 hybrid_clock.cc:648] HybridClock initialized: now 1714615075851364 us; error 35 us; skew 500 ppm
I20240502 01:57:55.853024 18392 webserver.cc:460] Webserver started at http://127.17.243.62:33811/ using document root <none> and password file <none>
I20240502 01:57:55.853484 18392 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:55.853540 18392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:55.853694 18392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:55.855144 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f1783df5a6ce4256b2a01992302e55f7"
format_stamp: "Formatted at 2024-05-02 01:57:55 on dist-test-slave-j3s5"
I20240502 01:57:55.855566 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f1783df5a6ce4256b2a01992302e55f7"
format_stamp: "Formatted at 2024-05-02 01:57:55 on dist-test-slave-j3s5"
I20240502 01:57:55.858431 18392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:57:55.860301 18407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:55.861162 18392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:55.861263 18392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f1783df5a6ce4256b2a01992302e55f7"
format_stamp: "Formatted at 2024-05-02 01:57:55 on dist-test-slave-j3s5"
I20240502 01:57:55.861333 18392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:55.899544 18392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:55.900079 18392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:55.900211 18392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:55.906167 18392 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.62:38681
I20240502 01:57:55.906191 18459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.62:38681 every 8 connection(s)
I20240502 01:57:55.907003 18392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:57:55.908813 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 18392
I20240502 01:57:55.908938 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:57:55.909246 18460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:55.914040 18460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7: Bootstrap starting.
I20240502 01:57:55.915827 18460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:55.916435 18460 log.cc:824] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:55.917744 18460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7: No bootstrap required, opened a new log
I20240502 01:57:55.919927 18460 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1783df5a6ce4256b2a01992302e55f7" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 38681 } }
I20240502 01:57:55.920083 18460 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:55.920161 18460 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1783df5a6ce4256b2a01992302e55f7, State: Initialized, Role: FOLLOWER
I20240502 01:57:55.920507 18460 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1783df5a6ce4256b2a01992302e55f7" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 38681 } }
I20240502 01:57:55.920620 18460 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:55.920678 18460 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:55.920786 18460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:55.921419 18460 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1783df5a6ce4256b2a01992302e55f7" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 38681 } }
I20240502 01:57:55.921695 18460 leader_election.cc:302] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1783df5a6ce4256b2a01992302e55f7; no voters:
I20240502 01:57:55.921913 18460 leader_election.cc:288] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:55.922200 18465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:55.922363 18465 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [term 1 LEADER]: Becoming Leader. State: Replica: f1783df5a6ce4256b2a01992302e55f7, State: Running, Role: LEADER
I20240502 01:57:55.922607 18465 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1783df5a6ce4256b2a01992302e55f7" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 38681 } }
I20240502 01:57:55.922909 18460 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:55.924000 18466 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1783df5a6ce4256b2a01992302e55f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1783df5a6ce4256b2a01992302e55f7" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 38681 } } }
I20240502 01:57:55.924052 18467 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1783df5a6ce4256b2a01992302e55f7. Latest consensus state: current_term: 1 leader_uuid: "f1783df5a6ce4256b2a01992302e55f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1783df5a6ce4256b2a01992302e55f7" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 38681 } } }
I20240502 01:57:55.924182 18466 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:55.924484 18467 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:55.924628 18474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:57:55.926990 18474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:57:55.931354 18474 catalog_manager.cc:1347] Generated new cluster ID: 657855607e3240e09ec31e3ef478601b
I20240502 01:57:55.931411 18474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:57:55.937503 18474 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:57:55.938607 18474 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:57:55.947304 18474 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f1783df5a6ce4256b2a01992302e55f7: Generated new TSK 0
I20240502 01:57:55.947970 18474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:57:55.953094 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.1:0
--local_ip_for_outbound_sockets=127.17.243.1
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--builtin_ntp_servers=127.17.243.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 01:57:56.049970 18484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.050134 18484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.050174 18484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.053094 18484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.053185 18484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.1
I20240502 01:57:56.056639 18484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--never_fsync=true
--heap_profile_path=/tmp/kudu.18484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.1
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.057363 18484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.058351 18484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.065783 18489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.065959 18490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.066573 18492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.067082 18484 server_base.cc:1034] running on GCE node
I20240502 01:57:56.067447 18484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.068007 18484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.069159 18484 hybrid_clock.cc:648] HybridClock initialized: now 1714615076069143 us; error 29 us; skew 500 ppm
I20240502 01:57:56.071017 18484 webserver.cc:460] Webserver started at http://127.17.243.1:45235/ using document root <none> and password file <none>
I20240502 01:57:56.071631 18484 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.071719 18484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.071933 18484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.073863 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1765b603eb674a6a8652da771c2339c1"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.074388 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1765b603eb674a6a8652da771c2339c1"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.077788 18484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:56.079710 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.080600 18484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.080686 18484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1765b603eb674a6a8652da771c2339c1"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.080752 18484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.093559 18484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.094058 18484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.094167 18484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.094576 18484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.095330 18484 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.095376 18484 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.095409 18484 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.095430 18484 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.102981 18484 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.1:34615
I20240502 01:57:56.103032 18611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.1:34615 every 8 connection(s)
I20240502 01:57:56.103770 18484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:57:56.107594 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 18484
I20240502 01:57:56.107663 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:57:56.109576 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.2:0
--local_ip_for_outbound_sockets=127.17.243.2
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--builtin_ntp_servers=127.17.243.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:57:56.111593 18612 heartbeater.cc:347] Connected to a master server at 127.17.243.62:38681
I20240502 01:57:56.111773 18612 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.112202 18612 heartbeater.cc:511] Master 127.17.243.62:38681 requested a full tablet report, sending...
I20240502 01:57:56.113579 18424 ts_manager.cc:194] Registered new tserver with Master: 1765b603eb674a6a8652da771c2339c1 (127.17.243.1:34615)
I20240502 01:57:56.115157 18424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.1:50115
W20240502 01:57:56.209810 18615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.209981 18615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.210021 18615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.212898 18615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.212985 18615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.2
I20240502 01:57:56.216380 18615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--never_fsync=true
--heap_profile_path=/tmp/kudu.18615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.2
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.217013 18615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.217885 18615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.224855 18620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.225097 18621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.225405 18623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.226092 18615 server_base.cc:1034] running on GCE node
I20240502 01:57:56.226418 18615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.226866 18615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.228003 18615 hybrid_clock.cc:648] HybridClock initialized: now 1714615076227991 us; error 24 us; skew 500 ppm
I20240502 01:57:56.229430 18615 webserver.cc:460] Webserver started at http://127.17.243.2:37963/ using document root <none> and password file <none>
I20240502 01:57:56.229887 18615 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.229964 18615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.230134 18615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.231549 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "51fac1f2d04f45a5bfa06b29358197ed"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.231956 18615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "51fac1f2d04f45a5bfa06b29358197ed"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.234766 18615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:57:56.236610 18629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.237429 18615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.237540 18615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51fac1f2d04f45a5bfa06b29358197ed"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.237607 18615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.258664 18615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.259140 18615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.259267 18615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.259701 18615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.260450 18615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.260494 18615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.260540 18615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.260565 18615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.268900 18615 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.2:42715
I20240502 01:57:56.268929 18743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.2:42715 every 8 connection(s)
I20240502 01:57:56.269660 18615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:57:56.274747 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 18615
I20240502 01:57:56.274848 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:57:56.276962 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.3:0
--local_ip_for_outbound_sockets=127.17.243.3
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--builtin_ntp_servers=127.17.243.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:57:56.277037 18744 heartbeater.cc:347] Connected to a master server at 127.17.243.62:38681
I20240502 01:57:56.277215 18744 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.277693 18744 heartbeater.cc:511] Master 127.17.243.62:38681 requested a full tablet report, sending...
I20240502 01:57:56.278642 18424 ts_manager.cc:194] Registered new tserver with Master: 51fac1f2d04f45a5bfa06b29358197ed (127.17.243.2:42715)
I20240502 01:57:56.279142 18424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.2:37421
W20240502 01:57:56.374939 18747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.375097 18747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.375149 18747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.378033 18747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.378137 18747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.3
I20240502 01:57:56.381541 18747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--never_fsync=true
--heap_profile_path=/tmp/kudu.18747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.3
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.382323 18747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.383294 18747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.390707 18753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.390964 18755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.391490 18752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.391984 18747 server_base.cc:1034] running on GCE node
I20240502 01:57:56.392259 18747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.392678 18747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.393805 18747 hybrid_clock.cc:648] HybridClock initialized: now 1714615076393799 us; error 23 us; skew 500 ppm
I20240502 01:57:56.395355 18747 webserver.cc:460] Webserver started at http://127.17.243.3:38705/ using document root <none> and password file <none>
I20240502 01:57:56.395810 18747 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.395860 18747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.396016 18747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.397388 18747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0f98acb739ae48879a85201092059aa6"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.397801 18747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0f98acb739ae48879a85201092059aa6"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.400653 18747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 01:57:56.402587 18761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.403548 18747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.403654 18747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0f98acb739ae48879a85201092059aa6"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.403741 18747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.418419 18747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.418977 18747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.419126 18747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.419597 18747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.420331 18747 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.420377 18747 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.420431 18747 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.420475 18747 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.428719 18747 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.3:33573
I20240502 01:57:56.428748 18874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.3:33573 every 8 connection(s)
I20240502 01:57:56.429503 18747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:57:56.431747 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 18747
I20240502 01:57:56.431840 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:57:56.433867 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.4:0
--local_ip_for_outbound_sockets=127.17.243.4
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--builtin_ntp_servers=127.17.243.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:57:56.437723 18875 heartbeater.cc:347] Connected to a master server at 127.17.243.62:38681
I20240502 01:57:56.437896 18875 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.438338 18875 heartbeater.cc:511] Master 127.17.243.62:38681 requested a full tablet report, sending...
I20240502 01:57:56.439208 18424 ts_manager.cc:194] Registered new tserver with Master: 0f98acb739ae48879a85201092059aa6 (127.17.243.3:33573)
I20240502 01:57:56.439699 18424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.3:48651
W20240502 01:57:56.531620 18878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.531769 18878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.531806 18878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.534674 18878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.534760 18878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.4
I20240502 01:57:56.538120 18878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--never_fsync=true
--heap_profile_path=/tmp/kudu.18878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.4
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.538739 18878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.539599 18878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.546613 18884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.546912 18886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.547260 18883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.547900 18878 server_base.cc:1034] running on GCE node
I20240502 01:57:56.548184 18878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.548633 18878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.549777 18878 hybrid_clock.cc:648] HybridClock initialized: now 1714615076549766 us; error 24 us; skew 500 ppm
I20240502 01:57:56.551304 18878 webserver.cc:460] Webserver started at http://127.17.243.4:45659/ using document root <none> and password file <none>
I20240502 01:57:56.551759 18878 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.551815 18878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.551960 18878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.553335 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c607ff86aca44abb82b3c1f478379491"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.553746 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c607ff86aca44abb82b3c1f478379491"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.556535 18878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:57:56.558393 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.559212 18878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.559311 18878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c607ff86aca44abb82b3c1f478379491"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.559379 18878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.571681 18878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.572196 18878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.572347 18878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.572865 18878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.573607 18878 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.573653 18878 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.573698 18878 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.573726 18878 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.581701 18878 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.4:43965
I20240502 01:57:56.581727 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.4:43965 every 8 connection(s)
I20240502 01:57:56.582424 18878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:57:56.588805 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 18878
I20240502 01:57:56.588922 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:57:56.590409 19006 heartbeater.cc:347] Connected to a master server at 127.17.243.62:38681
I20240502 01:57:56.590591 19006 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.590957 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.5:0
--local_ip_for_outbound_sockets=127.17.243.5
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--builtin_ntp_servers=127.17.243.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:57:56.591030 19006 heartbeater.cc:511] Master 127.17.243.62:38681 requested a full tablet report, sending...
I20240502 01:57:56.591914 18424 ts_manager.cc:194] Registered new tserver with Master: c607ff86aca44abb82b3c1f478379491 (127.17.243.4:43965)
I20240502 01:57:56.592420 18424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.4:45619
W20240502 01:57:56.690735 19009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.690896 19009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.690937 19009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.693828 19009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.693948 19009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.5
I20240502 01:57:56.697364 19009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:38681
--never_fsync=true
--heap_profile_path=/tmp/kudu.19009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.5
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.698060 19009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.699010 19009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.706223 19014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.706564 19015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.706872 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.707501 19009 server_base.cc:1034] running on GCE node
I20240502 01:57:56.707836 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.708351 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.709497 19009 hybrid_clock.cc:648] HybridClock initialized: now 1714615076709484 us; error 24 us; skew 500 ppm
I20240502 01:57:56.711005 19009 webserver.cc:460] Webserver started at http://127.17.243.5:46353/ using document root <none> and password file <none>
I20240502 01:57:56.711479 19009 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.711531 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.711709 19009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.713109 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3fcf3131e3c54eb29462315e2cfb0b1a"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.713523 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3fcf3131e3c54eb29462315e2cfb0b1a"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.716339 19009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:56.718189 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.719056 19009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.719148 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3fcf3131e3c54eb29462315e2cfb0b1a"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-j3s5"
I20240502 01:57:56.719218 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.750741 19009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.751235 19009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.751360 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.751811 19009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.752591 19009 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.752635 19009 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.752681 19009 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.752708 19009 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.760493 19009 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.5:45497
I20240502 01:57:56.760520 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.5:45497 every 8 connection(s)
I20240502 01:57:56.761243 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:57:56.765895 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 19009
I20240502 01:57:56.766018 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:57:56.769187 19137 heartbeater.cc:347] Connected to a master server at 127.17.243.62:38681
I20240502 01:57:56.769359 19137 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.769781 19137 heartbeater.cc:511] Master 127.17.243.62:38681 requested a full tablet report, sending...
I20240502 01:57:56.770610 18424 ts_manager.cc:194] Registered new tserver with Master: 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5:45497)
I20240502 01:57:56.771117 18424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.5:50073
I20240502 01:57:56.780057 18380 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:57:56.790045 18424 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34308:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:57:56.805359 18678 tablet_service.cc:1467] Processing CreateTablet for tablet d5b899e266ce4e82937721d1c1d75027 (DEFAULT_TABLE table=TestTable [id=1d0f2829623140a38eeae8534479f24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:56.805356 18809 tablet_service.cc:1467] Processing CreateTablet for tablet d5b899e266ce4e82937721d1c1d75027 (DEFAULT_TABLE table=TestTable [id=1d0f2829623140a38eeae8534479f24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:56.806216 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b899e266ce4e82937721d1c1d75027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:56.806216 18678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b899e266ce4e82937721d1c1d75027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:56.806557 18546 tablet_service.cc:1467] Processing CreateTablet for tablet d5b899e266ce4e82937721d1c1d75027 (DEFAULT_TABLE table=TestTable [id=1d0f2829623140a38eeae8534479f24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:57:56.807318 18546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b899e266ce4e82937721d1c1d75027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:56.810899 19154 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Bootstrap starting.
I20240502 01:57:56.811666 19155 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: Bootstrap starting.
I20240502 01:57:56.812731 19156 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Bootstrap starting.
I20240502 01:57:56.812970 19154 tablet_bootstrap.cc:654] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:56.813612 19155 tablet_bootstrap.cc:654] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:56.813736 19154 log.cc:824] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:56.814384 19155 log.cc:824] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:56.814553 19156 tablet_bootstrap.cc:654] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:56.815310 19154 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: No bootstrap required, opened a new log
I20240502 01:57:56.815304 19156 log.cc:824] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:56.815474 19154 ts_tablet_manager.cc:1367] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:57:56.815848 19155 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: No bootstrap required, opened a new log
I20240502 01:57:56.815995 19155 ts_tablet_manager.cc:1367] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:57:56.816818 19156 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: No bootstrap required, opened a new log
I20240502 01:57:56.816959 19156 ts_tablet_manager.cc:1367] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:57:56.818351 19154 raft_consensus.cc:348] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.818547 19154 raft_consensus.cc:374] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:56.818502 19155 raft_consensus.cc:348] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.818632 19154 raft_consensus.cc:729] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fac1f2d04f45a5bfa06b29358197ed, State: Initialized, Role: FOLLOWER
I20240502 01:57:56.818681 19155 raft_consensus.cc:374] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:56.818761 19155 raft_consensus.cc:729] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f98acb739ae48879a85201092059aa6, State: Initialized, Role: FOLLOWER
I20240502 01:57:56.819039 19154 consensus_queue.cc:260] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.819129 19155 consensus_queue.cc:260] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.819819 19154 ts_tablet_manager.cc:1398] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240502 01:57:56.819823 18744 heartbeater.cc:503] Master 127.17.243.62:38681 was elected leader, sending a full tablet report...
I20240502 01:57:56.819797 19156 raft_consensus.cc:348] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.819975 19156 raft_consensus.cc:374] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:56.820026 19156 raft_consensus.cc:729] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1765b603eb674a6a8652da771c2339c1, State: Initialized, Role: FOLLOWER
I20240502 01:57:56.820390 19156 consensus_queue.cc:260] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.820475 19155 ts_tablet_manager.cc:1398] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:57:56.820753 18875 heartbeater.cc:503] Master 127.17.243.62:38681 was elected leader, sending a full tablet report...
I20240502 01:57:56.821094 19156 ts_tablet_manager.cc:1398] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240502 01:57:56.822058 18612 heartbeater.cc:503] Master 127.17.243.62:38681 was elected leader, sending a full tablet report...
W20240502 01:57:56.854830 18613 tablet.cc:2369] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:57:56.930548 18876 tablet.cc:2369] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:56.954746 19162 raft_consensus.cc:482] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:57:56.954890 19162 raft_consensus.cc:504] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.955818 19162 leader_election.cc:288] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f98acb739ae48879a85201092059aa6 (127.17.243.3:33573), 51fac1f2d04f45a5bfa06b29358197ed (127.17.243.2:42715)
I20240502 01:57:56.959335 18698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027" candidate_uuid: "1765b603eb674a6a8652da771c2339c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fac1f2d04f45a5bfa06b29358197ed" is_pre_election: true
I20240502 01:57:56.959457 18829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027" candidate_uuid: "1765b603eb674a6a8652da771c2339c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f98acb739ae48879a85201092059aa6" is_pre_election: true
I20240502 01:57:56.959609 18698 raft_consensus.cc:2454] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1765b603eb674a6a8652da771c2339c1 in term 0.
I20240502 01:57:56.959663 18829 raft_consensus.cc:2454] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1765b603eb674a6a8652da771c2339c1 in term 0.
I20240502 01:57:56.960062 18500 leader_election.cc:302] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1765b603eb674a6a8652da771c2339c1, 51fac1f2d04f45a5bfa06b29358197ed; no voters:
I20240502 01:57:56.960314 19162 raft_consensus.cc:2781] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:57:56.960395 19162 raft_consensus.cc:482] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:57:56.960465 19162 raft_consensus.cc:3037] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:56.961324 19162 raft_consensus.cc:504] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.961663 19162 leader_election.cc:288] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [CANDIDATE]: Term 1 election: Requested vote from peers 0f98acb739ae48879a85201092059aa6 (127.17.243.3:33573), 51fac1f2d04f45a5bfa06b29358197ed (127.17.243.2:42715)
I20240502 01:57:56.961992 18829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027" candidate_uuid: "1765b603eb674a6a8652da771c2339c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f98acb739ae48879a85201092059aa6"
I20240502 01:57:56.962023 18698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027" candidate_uuid: "1765b603eb674a6a8652da771c2339c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fac1f2d04f45a5bfa06b29358197ed"
I20240502 01:57:56.962114 18698 raft_consensus.cc:3037] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:56.962103 18829 raft_consensus.cc:3037] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:56.962952 18698 raft_consensus.cc:2454] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1765b603eb674a6a8652da771c2339c1 in term 1.
I20240502 01:57:56.963009 18829 raft_consensus.cc:2454] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1765b603eb674a6a8652da771c2339c1 in term 1.
I20240502 01:57:56.963234 18500 leader_election.cc:302] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1765b603eb674a6a8652da771c2339c1, 51fac1f2d04f45a5bfa06b29358197ed; no voters:
I20240502 01:57:56.963388 19162 raft_consensus.cc:2781] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:56.963577 19162 raft_consensus.cc:686] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1765b603eb674a6a8652da771c2339c1, State: Running, Role: LEADER
I20240502 01:57:56.963830 19162 consensus_queue.cc:237] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } }
I20240502 01:57:56.966578 18422 catalog_manager.cc:5484] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1765b603eb674a6a8652da771c2339c1 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "1765b603eb674a6a8652da771c2339c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } health_report { overall_health: HEALTHY } } }
I20240502 01:57:56.985131 18380 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:57:56.986605 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1765b603eb674a6a8652da771c2339c1 to finish bootstrapping
I20240502 01:57:56.991461 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 51fac1f2d04f45a5bfa06b29358197ed to finish bootstrapping
I20240502 01:57:56.995668 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0f98acb739ae48879a85201092059aa6 to finish bootstrapping
I20240502 01:57:56.999569 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c607ff86aca44abb82b3c1f478379491 to finish bootstrapping
I20240502 01:57:57.004630 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3fcf3131e3c54eb29462315e2cfb0b1a to finish bootstrapping
W20240502 01:57:57.020870 18745 tablet.cc:2369] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:57:57.186978 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.187203 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.187662 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.187716 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.188417 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.392509 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.392935 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.392975 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.393453 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.394531 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.476722 19233 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:57.481590 19238 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:57:57.594539 19006 heartbeater.cc:503] Master 127.17.243.62:38681 was elected leader, sending a full tablet report...
I20240502 01:57:57.599555 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.599574 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.599745 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.607838 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.609799 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.772673 19137 heartbeater.cc:503] Master 127.17.243.62:38681 was elected leader, sending a full tablet report...
I20240502 01:57:57.804756 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.805174 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.805373 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.805873 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.817305 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.890010 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.891546 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.891954 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.892743 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.892743 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.987494 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.988968 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.992290 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.994107 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.994349 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.149099 18422 ts_manager.cc:292] Set tserver state for 1765b603eb674a6a8652da771c2339c1 to MAINTENANCE_MODE
I20240502 01:57:58.316975 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.316970 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.317271 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.317272 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.322260 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.395331 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.396109 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.396320 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.397315 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.398083 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.495688 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.496794 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.497160 18678 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.497911 18546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.499971 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.534257 18566 consensus_queue.cc:237] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } }
I20240502 01:57:58.535485 18698 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Refusing update from remote peer 1765b603eb674a6a8652da771c2339c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.535521 18829 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 1 FOLLOWER]: Refusing update from remote peer 1765b603eb674a6a8652da771c2339c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:57:58.535887 19238 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:57:58.536053 19373 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:57:58.537773 19238 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } }
I20240502 01:57:58.538239 18698 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } }
I20240502 01:57:58.538193 18829 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } }
I20240502 01:57:58.540154 18422 catalog_manager.cc:5484] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "1765b603eb674a6a8652da771c2339c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240502 01:57:58.543656 18566 consensus_queue.cc:237] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } }
I20240502 01:57:58.544860 18829 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 1 FOLLOWER]: Refusing update from remote peer 1765b603eb674a6a8652da771c2339c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:57:58.544940 18698 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Refusing update from remote peer 1765b603eb674a6a8652da771c2339c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:57:58.545264 19238 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:57:58.545437 19372 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:57:58.546954 19373 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } } }
I20240502 01:57:58.547466 18829 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } } }
I20240502 01:57:58.547461 18698 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } } }
I20240502 01:57:58.549340 18409 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d5b899e266ce4e82937721d1c1d75027 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:57:58.549671 18422 catalog_manager.cc:5484] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 reported cstate change: config changed from index 2 to 3, NON_VOTER 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "1765b603eb674a6a8652da771c2339c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:57:58.552294 18501 consensus_peers.cc:485] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 -> Peer 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5:45497): Couldn't send request to peer 3fcf3131e3c54eb29462315e2cfb0b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5b899e266ce4e82937721d1c1d75027. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:57:58.552714 18566 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027"
dest_uuid: "1765b603eb674a6a8652da771c2339c1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41648
I20240502 01:57:58.552842 18566 raft_consensus.cc:595] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:57:58.963004 19238 raft_consensus.cc:979] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: : Instructing follower 51fac1f2d04f45a5bfa06b29358197ed to start an election
I20240502 01:57:58.963152 19372 raft_consensus.cc:1067] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Signalling peer 51fac1f2d04f45a5bfa06b29358197ed to start an election
I20240502 01:57:58.963835 18698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027"
dest_uuid: "51fac1f2d04f45a5bfa06b29358197ed"
from {username='slave'} at 127.17.243.1:55439
I20240502 01:57:58.964042 18698 raft_consensus.cc:482] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:57:58.964107 18698 raft_consensus.cc:3037] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:58.965135 18698 raft_consensus.cc:504] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } }
I20240502 01:57:58.966224 18698 leader_election.cc:288] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [CANDIDATE]: Term 2 election: Requested vote from peers 0f98acb739ae48879a85201092059aa6 (127.17.243.3:33573), 1765b603eb674a6a8652da771c2339c1 (127.17.243.1:34615)
I20240502 01:57:58.969739 18829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027" candidate_uuid: "51fac1f2d04f45a5bfa06b29358197ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f98acb739ae48879a85201092059aa6"
I20240502 01:57:58.969738 18566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b899e266ce4e82937721d1c1d75027" candidate_uuid: "51fac1f2d04f45a5bfa06b29358197ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1765b603eb674a6a8652da771c2339c1"
I20240502 01:57:58.969880 18829 raft_consensus.cc:3037] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:58.969918 18566 raft_consensus.cc:3032] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:57:58.969998 18566 raft_consensus.cc:729] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1765b603eb674a6a8652da771c2339c1, State: Running, Role: LEADER
I20240502 01:57:58.970181 18566 consensus_queue.cc:260] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } }
I20240502 01:57:58.970425 18566 raft_consensus.cc:3037] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:57:58.970924 18829 raft_consensus.cc:2454] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fac1f2d04f45a5bfa06b29358197ed in term 2.
I20240502 01:57:58.971333 18630 leader_election.cc:302] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f98acb739ae48879a85201092059aa6, 51fac1f2d04f45a5bfa06b29358197ed; no voters:
I20240502 01:57:58.971501 18566 raft_consensus.cc:2454] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fac1f2d04f45a5bfa06b29358197ed in term 2.
I20240502 01:57:58.971614 19494 raft_consensus.cc:2781] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:57:58.971793 19494 raft_consensus.cc:686] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 2 LEADER]: Becoming Leader. State: Replica: 51fac1f2d04f45a5bfa06b29358197ed, State: Running, Role: LEADER
I20240502 01:57:58.972090 19494 consensus_queue.cc:237] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } }
I20240502 01:57:58.974367 18423 catalog_manager.cc:5484] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed reported cstate change: term changed from 1 to 2, leader changed from 1765b603eb674a6a8652da771c2339c1 (127.17.243.1) to 51fac1f2d04f45a5bfa06b29358197ed (127.17.243.2). New cstate: current_term: 2 leader_uuid: "51fac1f2d04f45a5bfa06b29358197ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:57:59.523582 18829 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 2 FOLLOWER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:59.524098 19530 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:57:59.527246 18566 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 2 FOLLOWER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:57:59.528818 19532 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:57:59.531630 18632 consensus_peers.cc:485] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed -> Peer 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5:45497): Couldn't send request to peer 3fcf3131e3c54eb29462315e2cfb0b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5b899e266ce4e82937721d1c1d75027. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:00.024920 19555 ts_tablet_manager.cc:902] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Initiating tablet copy from peer 51fac1f2d04f45a5bfa06b29358197ed (127.17.243.2:42715)
I20240502 01:58:00.025609 19555 tablet_copy_client.cc:308] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:42715
I20240502 01:58:00.029244 18718 tablet_copy_service.cc:140] P 51fac1f2d04f45a5bfa06b29358197ed: Received BeginTabletCopySession request for tablet d5b899e266ce4e82937721d1c1d75027 from peer 3fcf3131e3c54eb29462315e2cfb0b1a ({username='slave'} at 127.17.243.5:49327)
I20240502 01:58:00.029361 18718 tablet_copy_service.cc:161] P 51fac1f2d04f45a5bfa06b29358197ed: Beginning new tablet copy session on tablet d5b899e266ce4e82937721d1c1d75027 from peer 3fcf3131e3c54eb29462315e2cfb0b1a at {username='slave'} at 127.17.243.5:49327: session id = 3fcf3131e3c54eb29462315e2cfb0b1a-d5b899e266ce4e82937721d1c1d75027
I20240502 01:58:00.030997 18718 tablet_copy_source_session.cc:205] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:00.033821 19555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b899e266ce4e82937721d1c1d75027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:00.038619 19555 tablet_copy_client.cc:783] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:00.038797 19555 tablet_copy_client.cc:647] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:00.040205 19555 tablet_copy_client.cc:523] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:00.041558 19555 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Bootstrap starting.
I20240502 01:58:00.045751 19555 log.cc:824] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:00.048094 19555 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:00.048291 19555 tablet_bootstrap.cc:492] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Bootstrap complete.
I20240502 01:58:00.048493 19555 ts_tablet_manager.cc:1367] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 01:58:00.050761 19555 raft_consensus.cc:348] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } }
I20240502 01:58:00.050945 19555 raft_consensus.cc:729] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3fcf3131e3c54eb29462315e2cfb0b1a, State: Initialized, Role: LEARNER
I20240502 01:58:00.051216 19555 consensus_queue.cc:260] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: true } }
I20240502 01:58:00.051908 19555 ts_tablet_manager.cc:1398] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:00.052455 18718 tablet_copy_service.cc:342] P 51fac1f2d04f45a5bfa06b29358197ed: Request end of tablet copy session 3fcf3131e3c54eb29462315e2cfb0b1a-d5b899e266ce4e82937721d1c1d75027 received from {username='slave'} at 127.17.243.5:49327
I20240502 01:58:00.052546 18718 tablet_copy_service.cc:433] P 51fac1f2d04f45a5bfa06b29358197ed: ending tablet copy session 3fcf3131e3c54eb29462315e2cfb0b1a-d5b899e266ce4e82937721d1c1d75027 on tablet d5b899e266ce4e82937721d1c1d75027 with peer 3fcf3131e3c54eb29462315e2cfb0b1a
I20240502 01:58:00.478381 19089 raft_consensus.cc:1203] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 LEARNER]: Deduplicated request from leader. Original: 1.3->[2.4-2.4] Dedup: 2.4->[]
I20240502 01:58:01.100976 19532 raft_consensus.cc:1050] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed: attempting to promote NON_VOTER 3fcf3131e3c54eb29462315e2cfb0b1a to VOTER
I20240502 01:58:01.101649 19532 consensus_queue.cc:237] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } }
I20240502 01:58:01.102967 19089 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 LEARNER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:58:01.103152 18566 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 2 FOLLOWER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:58:01.103142 18829 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 2 FOLLOWER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:58:01.103610 19588 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:01.103849 19591 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:01.104041 19590 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:01.106820 19588 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.107312 19089 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.108755 18829 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.109268 18423 catalog_manager.cc:5484] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed reported cstate change: config changed from index 3 to 5, 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "51fac1f2d04f45a5bfa06b29358197ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:01.110987 18566 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 3fcf3131e3c54eb29462315e2cfb0b1a (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "1765b603eb674a6a8652da771c2339c1" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 34615 } attrs { replace: true } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.113561 18698 consensus_queue.cc:237] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } }
I20240502 01:58:01.114486 19089 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 FOLLOWER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:01.114567 18829 raft_consensus.cc:1261] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 2 FOLLOWER]: Refusing update from remote peer 51fac1f2d04f45a5bfa06b29358197ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:01.114820 19591 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:01.115191 19591 consensus_queue.cc:1035] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:01.116559 19590 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 51fac1f2d04f45a5bfa06b29358197ed [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 1765b603eb674a6a8652da771c2339c1 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.116839 18829 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 1765b603eb674a6a8652da771c2339c1 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.116853 19089 raft_consensus.cc:2932] T d5b899e266ce4e82937721d1c1d75027 P 3fcf3131e3c54eb29462315e2cfb0b1a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 1765b603eb674a6a8652da771c2339c1 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.118268 18409 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5b899e266ce4e82937721d1c1d75027 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:01.118535 18422 catalog_manager.cc:5484] T d5b899e266ce4e82937721d1c1d75027 P 0f98acb739ae48879a85201092059aa6 reported cstate change: config changed from index 5 to 6, VOTER 1765b603eb674a6a8652da771c2339c1 (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "51fac1f2d04f45a5bfa06b29358197ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0f98acb739ae48879a85201092059aa6" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 33573 } } peers { permanent_uuid: "51fac1f2d04f45a5bfa06b29358197ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 42715 } } peers { permanent_uuid: "3fcf3131e3c54eb29462315e2cfb0b1a" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 45497 } attrs { promote: false } } }
I20240502 01:58:01.120730 18546 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b899e266ce4e82937721d1c1d75027 with delete_type TABLET_DATA_TOMBSTONED (TS 1765b603eb674a6a8652da771c2339c1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55268
I20240502 01:58:01.121104 19602 tablet_replica.cc:330] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: stopping tablet replica
I20240502 01:58:01.121336 19602 raft_consensus.cc:2229] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:01.121480 19602 raft_consensus.cc:2258] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:01.122447 19602 ts_tablet_manager.cc:1875] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:01.124172 19602 ts_tablet_manager.cc:1888] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:58:01.124243 19602 log.cc:1196] T d5b899e266ce4e82937721d1c1d75027 P 1765b603eb674a6a8652da771c2339c1: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/d5b899e266ce4e82937721d1c1d75027
I20240502 01:58:01.124804 18409 catalog_manager.cc:4830] TS 1765b603eb674a6a8652da771c2339c1 (127.17.243.1:34615): tablet d5b899e266ce4e82937721d1c1d75027 (table TestTable [id=1d0f2829623140a38eeae8534479f24a]) successfully deleted
I20240502 01:58:01.475741 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.477437 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.478829 18678 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:01.482002 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.485038 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.552812 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.554174 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.555096 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.556566 18678 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:01.557247 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.629149 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.629787 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.631050 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.634397 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.650606 18678 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:01.673005 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 18615
I20240502 01:58:01.842975 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.843051 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.844686 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.850106 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.910634 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.912827 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.914381 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.921972 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.007370 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.009150 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.010087 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.010488 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.083168 18809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.083460 19071 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.086287 18940 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.096334 18546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.132320 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 18484
I20240502 01:58:02.138352 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 18747
I20240502 01:58:02.143566 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 18878
I20240502 01:58:02.147956 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 19009
I20240502 01:58:02.152302 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 18392
2024-05-02T01:58:02Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (6453 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (6453 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240502 01:58:02.172487 18380 test_util.cc:268] Using random seed: -286900156
I20240502 01:58:02.173111 18380 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:58:02.173142 18380 ts_itest-base.cc:113] --------------
I20240502 01:58:02.173151 18380 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:58:02.173161 18380 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:58:02.173169 18380 ts_itest-base.cc:116] --------------
2024-05-02T01:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:58:02Z Disabled control of system clock
I20240502 01:58:02.179957 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:33101
--webserver_interface=127.17.243.62
--webserver_port=0
--builtin_ntp_servers=127.17.243.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.243.62:33101
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:58:02.275857 19822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:02.276031 19822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:02.276067 19822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:02.278883 19822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:02.278934 19822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:58:02.278951 19822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:02.278975 19822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:58:02.278993 19822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:58:02.282392 19822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.243.62:33101
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.243.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:02.283051 19822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:02.284000 19822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:02.291111 19822 server_base.cc:1034] running on GCE node
W20240502 01:58:02.291577 19827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.292023 19828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.292398 19830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:02.292886 19822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:02.293737 19822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:02.294884 19822 hybrid_clock.cc:648] HybridClock initialized: now 1714615082294869 us; error 32 us; skew 500 ppm
I20240502 01:58:02.300536 19822 webserver.cc:460] Webserver started at http://127.17.243.62:44987/ using document root <none> and password file <none>
I20240502 01:58:02.301035 19822 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:02.301087 19822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:02.301247 19822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:02.302846 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "96d9879012854904b01b9f50c27396f3"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.303282 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "96d9879012854904b01b9f50c27396f3"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.306373 19822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:02.308368 19836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.309305 19822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:02.309407 19822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
uuid: "96d9879012854904b01b9f50c27396f3"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.309494 19822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:02.328424 19822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:02.328934 19822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:02.329068 19822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:02.335085 19822 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.62:33101
I20240502 01:58:02.335105 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.62:33101 every 8 connection(s)
I20240502 01:58:02.335922 19822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:58:02.338389 19889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.343010 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3: Bootstrap starting.
I20240502 01:58:02.344038 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 19822
I20240502 01:58:02.344149 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:58:02.345320 19889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.346047 19889 log.cc:824] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:02.347484 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3: No bootstrap required, opened a new log
I20240502 01:58:02.349829 19889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9879012854904b01b9f50c27396f3" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33101 } }
I20240502 01:58:02.350033 19889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.350106 19889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d9879012854904b01b9f50c27396f3, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.350497 19889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9879012854904b01b9f50c27396f3" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33101 } }
I20240502 01:58:02.350652 19889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:02.350716 19889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:02.350836 19889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.351660 19889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9879012854904b01b9f50c27396f3" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33101 } }
I20240502 01:58:02.351962 19889 leader_election.cc:302] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96d9879012854904b01b9f50c27396f3; no voters:
I20240502 01:58:02.352185 19889 leader_election.cc:288] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:02.352272 19894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.352504 19894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [term 1 LEADER]: Becoming Leader. State: Replica: 96d9879012854904b01b9f50c27396f3, State: Running, Role: LEADER
I20240502 01:58:02.352775 19894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9879012854904b01b9f50c27396f3" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33101 } }
I20240502 01:58:02.353152 19889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:02.353698 19896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96d9879012854904b01b9f50c27396f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9879012854904b01b9f50c27396f3" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33101 } } }
I20240502 01:58:02.354064 19895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d9879012854904b01b9f50c27396f3. Latest consensus state: current_term: 1 leader_uuid: "96d9879012854904b01b9f50c27396f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9879012854904b01b9f50c27396f3" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33101 } } }
I20240502 01:58:02.354108 19896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:02.354135 19895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:02.354475 19899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:02.357110 19899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:02.362128 19899 catalog_manager.cc:1347] Generated new cluster ID: 0cd0858090bb43ff9025ab25e52d0d2a
I20240502 01:58:02.362186 19899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:02.372109 19899 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:02.372938 19899 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:02.379408 19899 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 96d9879012854904b01b9f50c27396f3: Generated new TSK 0
I20240502 01:58:02.379952 19899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:02.383934 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.1:0
--local_ip_for_outbound_sockets=127.17.243.1
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--builtin_ntp_servers=127.17.243.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:02.491878 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:02.492072 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:02.492137 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:02.495023 19913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:02.495088 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:02.495172 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.1
I20240502 01:58:02.498579 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.1
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:02.499302 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:02.500279 19913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:02.508348 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.508587 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.509195 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:02.509841 19913 server_base.cc:1034] running on GCE node
I20240502 01:58:02.510178 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:02.510633 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:02.511783 19913 hybrid_clock.cc:648] HybridClock initialized: now 1714615082511766 us; error 31 us; skew 500 ppm
I20240502 01:58:02.513485 19913 webserver.cc:460] Webserver started at http://127.17.243.1:40867/ using document root <none> and password file <none>
I20240502 01:58:02.514014 19913 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:02.514070 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:02.514236 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:02.515708 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.516129 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.519102 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:02.520987 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.521885 19913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:02.522008 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.522082 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:02.535598 19913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:02.536067 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:02.536193 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:02.536623 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:02.537379 19913 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:02.537425 19913 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.537470 19913 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:02.537498 19913 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.545329 19913 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.1:43885
I20240502 01:58:02.545353 20040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.1:43885 every 8 connection(s)
I20240502 01:58:02.546108 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:58:02.549063 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 19913
I20240502 01:58:02.549156 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:58:02.551137 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.2:0
--local_ip_for_outbound_sockets=127.17.243.2
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--builtin_ntp_servers=127.17.243.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:02.554229 20041 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33101
I20240502 01:58:02.554421 20041 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:02.554919 20041 heartbeater.cc:511] Master 127.17.243.62:33101 requested a full tablet report, sending...
I20240502 01:58:02.556357 19850 ts_manager.cc:194] Registered new tserver with Master: b5c209d86d3a4f459d8b8f3ee5cab66f (127.17.243.1:43885)
I20240502 01:58:02.558036 19850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.1:45771
W20240502 01:58:02.653035 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:02.653198 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:02.653236 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:02.656073 20044 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:02.656124 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:02.656201 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.2
I20240502 01:58:02.659508 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--never_fsync=true
--heap_profile_path=/tmp/kudu.20044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.2
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:02.660145 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:02.661039 20044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:02.668481 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.668748 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.669411 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:02.670068 20044 server_base.cc:1034] running on GCE node
I20240502 01:58:02.670365 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:02.670825 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:02.671967 20044 hybrid_clock.cc:648] HybridClock initialized: now 1714615082671954 us; error 27 us; skew 500 ppm
I20240502 01:58:02.673626 20044 webserver.cc:460] Webserver started at http://127.17.243.2:35549/ using document root <none> and password file <none>
I20240502 01:58:02.674120 20044 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:02.674172 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:02.674331 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:02.675729 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ed66cf01c18543fe9305f45ddd947d34"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.676169 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ed66cf01c18543fe9305f45ddd947d34"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.678992 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:02.680816 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.681654 20044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:02.681749 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ed66cf01c18543fe9305f45ddd947d34"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.681820 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:02.713081 20044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:02.713508 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:02.713636 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:02.714123 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:02.714855 20044 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:02.714908 20044 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.714975 20044 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:02.715003 20044 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.722318 20044 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.2:41635
I20240502 01:58:02.722340 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.2:41635 every 8 connection(s)
I20240502 01:58:02.723045 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:58:02.726590 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 20044
I20240502 01:58:02.726711 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:58:02.728817 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.3:0
--local_ip_for_outbound_sockets=127.17.243.3
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--builtin_ntp_servers=127.17.243.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:02.731233 20172 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33101
I20240502 01:58:02.731406 20172 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:02.731830 20172 heartbeater.cc:511] Master 127.17.243.62:33101 requested a full tablet report, sending...
I20240502 01:58:02.732726 19850 ts_manager.cc:194] Registered new tserver with Master: ed66cf01c18543fe9305f45ddd947d34 (127.17.243.2:41635)
I20240502 01:58:02.733242 19850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.2:47805
W20240502 01:58:02.825750 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:02.825989 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:02.826047 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:02.828999 20175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:02.829053 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:02.829129 20175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.3
I20240502 01:58:02.832583 20175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.3
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:02.833389 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:02.834463 20175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:02.842310 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.842859 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:02.843226 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:02.843729 20175 server_base.cc:1034] running on GCE node
I20240502 01:58:02.844058 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:02.844535 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:02.845719 20175 hybrid_clock.cc:648] HybridClock initialized: now 1714615082845706 us; error 28 us; skew 500 ppm
I20240502 01:58:02.847626 20175 webserver.cc:460] Webserver started at http://127.17.243.3:44861/ using document root <none> and password file <none>
I20240502 01:58:02.848125 20175 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:02.848181 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:02.848343 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:02.849916 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e40a8be211714d24b0df48a4137bffb6"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.850432 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e40a8be211714d24b0df48a4137bffb6"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.853590 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:02.855960 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.856900 20175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:02.856998 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e40a8be211714d24b0df48a4137bffb6"
format_stamp: "Formatted at 2024-05-02 01:58:02 on dist-test-slave-j3s5"
I20240502 01:58:02.857071 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:02.879739 20175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:02.880266 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:02.880409 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:02.880905 20175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:02.881772 20175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:02.881824 20175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.881877 20175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:02.881909 20175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:02.890065 20175 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.3:34679
I20240502 01:58:02.890089 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.3:34679 every 8 connection(s)
I20240502 01:58:02.890882 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:58:02.893880 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 20175
I20240502 01:58:02.894003 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:58:02.896139 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.4:0
--local_ip_for_outbound_sockets=127.17.243.4
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--builtin_ntp_servers=127.17.243.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:02.902922 20303 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33101
I20240502 01:58:02.903329 20303 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:02.903970 20303 heartbeater.cc:511] Master 127.17.243.62:33101 requested a full tablet report, sending...
I20240502 01:58:02.904891 19850 ts_manager.cc:194] Registered new tserver with Master: e40a8be211714d24b0df48a4137bffb6 (127.17.243.3:34679)
I20240502 01:58:02.905400 19850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.3:39833
W20240502 01:58:02.999929 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:03.000113 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:03.000173 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:03.003103 20306 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:03.003175 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:03.003261 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.4
I20240502 01:58:03.006680 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--never_fsync=true
--heap_profile_path=/tmp/kudu.20306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.4
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:03.007401 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:03.008396 20306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:03.016291 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:03.016770 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:03.017313 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:03.017832 20306 server_base.cc:1034] running on GCE node
I20240502 01:58:03.018205 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:03.018687 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:03.019826 20306 hybrid_clock.cc:648] HybridClock initialized: now 1714615083019814 us; error 25 us; skew 500 ppm
I20240502 01:58:03.021486 20306 webserver.cc:460] Webserver started at http://127.17.243.4:37837/ using document root <none> and password file <none>
I20240502 01:58:03.021998 20306 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:03.022063 20306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:03.022239 20306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:03.023758 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bbe0c2501f074a368b609acea2e52b06"
format_stamp: "Formatted at 2024-05-02 01:58:03 on dist-test-slave-j3s5"
I20240502 01:58:03.024175 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bbe0c2501f074a368b609acea2e52b06"
format_stamp: "Formatted at 2024-05-02 01:58:03 on dist-test-slave-j3s5"
I20240502 01:58:03.027150 20306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 01:58:03.029021 20320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:03.029913 20306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:03.030037 20306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bbe0c2501f074a368b609acea2e52b06"
format_stamp: "Formatted at 2024-05-02 01:58:03 on dist-test-slave-j3s5"
I20240502 01:58:03.030114 20306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:03.045567 20306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:03.046087 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:03.046233 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:03.046697 20306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:03.047559 20306 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:03.047608 20306 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:03.047660 20306 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:03.047693 20306 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:03.056291 20306 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.4:42643
I20240502 01:58:03.056326 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.4:42643 every 8 connection(s)
I20240502 01:58:03.057083 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:03.062780 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 20306
I20240502 01:58:03.062902 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:58:03.065086 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.5:0
--local_ip_for_outbound_sockets=127.17.243.5
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--builtin_ntp_servers=127.17.243.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:03.065217 20434 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33101
I20240502 01:58:03.065445 20434 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:03.066084 20434 heartbeater.cc:511] Master 127.17.243.62:33101 requested a full tablet report, sending...
I20240502 01:58:03.067092 19850 ts_manager.cc:194] Registered new tserver with Master: bbe0c2501f074a368b609acea2e52b06 (127.17.243.4:42643)
I20240502 01:58:03.067624 19850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.4:33461
W20240502 01:58:03.164124 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:03.164311 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:03.164382 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:03.167265 20437 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:03.167331 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:03.167429 20437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.5
I20240502 01:58:03.170790 20437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33101
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.5
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:03.171527 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:03.172513 20437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:03.179890 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:03.180223 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:03.180716 20445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:03.181214 20437 server_base.cc:1034] running on GCE node
I20240502 01:58:03.181572 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:03.182065 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:03.183215 20437 hybrid_clock.cc:648] HybridClock initialized: now 1714615083183196 us; error 30 us; skew 500 ppm
I20240502 01:58:03.184887 20437 webserver.cc:460] Webserver started at http://127.17.243.5:35577/ using document root <none> and password file <none>
I20240502 01:58:03.185345 20437 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:03.185407 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:03.185577 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:03.187074 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "bb6c8ed6ab3746269ba7cdd5a4874579"
format_stamp: "Formatted at 2024-05-02 01:58:03 on dist-test-slave-j3s5"
I20240502 01:58:03.187515 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "bb6c8ed6ab3746269ba7cdd5a4874579"
format_stamp: "Formatted at 2024-05-02 01:58:03 on dist-test-slave-j3s5"
I20240502 01:58:03.190632 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:03.192607 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:03.193478 20437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:03.193579 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "bb6c8ed6ab3746269ba7cdd5a4874579"
format_stamp: "Formatted at 2024-05-02 01:58:03 on dist-test-slave-j3s5"
I20240502 01:58:03.193657 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:03.208261 20437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:03.208710 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:03.208853 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:03.209297 20437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:03.210106 20437 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:03.210166 20437 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:03.210218 20437 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:03.210247 20437 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:03.218647 20437 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.5:40091
I20240502 01:58:03.218741 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.5:40091 every 8 connection(s)
I20240502 01:58:03.219588 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:03.219894 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 20437
I20240502 01:58:03.219997 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:58:03.227355 20565 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33101
I20240502 01:58:03.227559 20565 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:03.228004 20565 heartbeater.cc:511] Master 127.17.243.62:33101 requested a full tablet report, sending...
I20240502 01:58:03.228885 19850 ts_manager.cc:194] Registered new tserver with Master: bb6c8ed6ab3746269ba7cdd5a4874579 (127.17.243.5:40091)
I20240502 01:58:03.229423 19850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.5:47491
I20240502 01:58:03.233841 18380 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:58:03.241609 19850 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57330:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:58:03.257689 19975 tablet_service.cc:1467] Processing CreateTablet for tablet be7c73395b8349d6b5256ba29154af73 (DEFAULT_TABLE table=TestTable [id=b69e244b44d54e778edb07c44a8230bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.257803 20499 tablet_service.cc:1467] Processing CreateTablet for tablet be7c73395b8349d6b5256ba29154af73 (DEFAULT_TABLE table=TestTable [id=b69e244b44d54e778edb07c44a8230bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.258545 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7c73395b8349d6b5256ba29154af73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.258602 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7c73395b8349d6b5256ba29154af73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.263135 20583 tablet_bootstrap.cc:492] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: Bootstrap starting.
I20240502 01:58:03.263567 20582 tablet_bootstrap.cc:492] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: Bootstrap starting.
I20240502 01:58:03.265061 20368 tablet_service.cc:1467] Processing CreateTablet for tablet be7c73395b8349d6b5256ba29154af73 (DEFAULT_TABLE table=TestTable [id=b69e244b44d54e778edb07c44a8230bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.265448 20583 tablet_bootstrap.cc:654] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.265448 20582 tablet_bootstrap.cc:654] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.265763 20368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7c73395b8349d6b5256ba29154af73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.266232 20583 log.cc:824] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:03.266232 20582 log.cc:824] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:03.267719 20583 tablet_bootstrap.cc:492] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: No bootstrap required, opened a new log
I20240502 01:58:03.267832 20582 tablet_bootstrap.cc:492] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: No bootstrap required, opened a new log
I20240502 01:58:03.267874 20583 ts_tablet_manager.cc:1367] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:58:03.268007 20582 ts_tablet_manager.cc:1367] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:58:03.270457 20583 raft_consensus.cc:348] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.270624 20583 raft_consensus.cc:374] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.270692 20583 raft_consensus.cc:729] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c209d86d3a4f459d8b8f3ee5cab66f, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.271179 20586 tablet_bootstrap.cc:492] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: Bootstrap starting.
I20240502 01:58:03.271145 20583 consensus_queue.cc:260] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.271646 20582 raft_consensus.cc:348] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.271821 20582 raft_consensus.cc:374] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.271915 20582 raft_consensus.cc:729] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6c8ed6ab3746269ba7cdd5a4874579, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.272064 20583 ts_tablet_manager.cc:1398] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240502 01:58:03.272296 20582 consensus_queue.cc:260] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.272457 20041 heartbeater.cc:503] Master 127.17.243.62:33101 was elected leader, sending a full tablet report...
I20240502 01:58:03.272938 20565 heartbeater.cc:503] Master 127.17.243.62:33101 was elected leader, sending a full tablet report...
I20240502 01:58:03.273690 20586 tablet_bootstrap.cc:654] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.274008 20582 ts_tablet_manager.cc:1398] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 01:58:03.274499 20586 log.cc:824] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:03.275864 20586 tablet_bootstrap.cc:492] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: No bootstrap required, opened a new log
I20240502 01:58:03.276007 20586 ts_tablet_manager.cc:1367] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:58:03.278746 20586 raft_consensus.cc:348] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.278890 20586 raft_consensus.cc:374] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.278939 20586 raft_consensus.cc:729] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe0c2501f074a368b609acea2e52b06, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.279276 20586 consensus_queue.cc:260] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.279868 20434 heartbeater.cc:503] Master 127.17.243.62:33101 was elected leader, sending a full tablet report...
I20240502 01:58:03.280005 20586 ts_tablet_manager.cc:1398] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20240502 01:58:03.297248 20042 tablet.cc:2369] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:03.307802 20435 tablet.cc:2369] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:03.356498 20587 raft_consensus.cc:482] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:03.356638 20587 raft_consensus.cc:504] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.357543 20587 leader_election.cc:288] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb6c8ed6ab3746269ba7cdd5a4874579 (127.17.243.5:40091), bbe0c2501f074a368b609acea2e52b06 (127.17.243.4:42643)
I20240502 01:58:03.361095 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7c73395b8349d6b5256ba29154af73" candidate_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" is_pre_election: true
I20240502 01:58:03.361145 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7c73395b8349d6b5256ba29154af73" candidate_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe0c2501f074a368b609acea2e52b06" is_pre_election: true
I20240502 01:58:03.361325 20519 raft_consensus.cc:2454] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c209d86d3a4f459d8b8f3ee5cab66f in term 0.
I20240502 01:58:03.361356 20388 raft_consensus.cc:2454] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c209d86d3a4f459d8b8f3ee5cab66f in term 0.
I20240502 01:58:03.361716 19930 leader_election.cc:302] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c209d86d3a4f459d8b8f3ee5cab66f, bb6c8ed6ab3746269ba7cdd5a4874579; no voters:
I20240502 01:58:03.361953 20587 raft_consensus.cc:2781] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:03.362023 20587 raft_consensus.cc:482] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:03.362069 20587 raft_consensus.cc:3037] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.362823 20587 raft_consensus.cc:504] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.363107 20587 leader_election.cc:288] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [CANDIDATE]: Term 1 election: Requested vote from peers bb6c8ed6ab3746269ba7cdd5a4874579 (127.17.243.5:40091), bbe0c2501f074a368b609acea2e52b06 (127.17.243.4:42643)
I20240502 01:58:03.363385 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7c73395b8349d6b5256ba29154af73" candidate_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe0c2501f074a368b609acea2e52b06"
I20240502 01:58:03.363385 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7c73395b8349d6b5256ba29154af73" candidate_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579"
I20240502 01:58:03.363476 20388 raft_consensus.cc:3037] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.363489 20519 raft_consensus.cc:3037] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.364223 20388 raft_consensus.cc:2454] T be7c73395b8349d6b5256ba29154af73 P bbe0c2501f074a368b609acea2e52b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c209d86d3a4f459d8b8f3ee5cab66f in term 1.
I20240502 01:58:03.364244 20519 raft_consensus.cc:2454] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c209d86d3a4f459d8b8f3ee5cab66f in term 1.
I20240502 01:58:03.364483 19928 leader_election.cc:302] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c209d86d3a4f459d8b8f3ee5cab66f, bbe0c2501f074a368b609acea2e52b06; no voters:
I20240502 01:58:03.364629 20587 raft_consensus.cc:2781] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:03.364795 20587 raft_consensus.cc:686] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [term 1 LEADER]: Becoming Leader. State: Replica: b5c209d86d3a4f459d8b8f3ee5cab66f, State: Running, Role: LEADER
I20240502 01:58:03.365074 20587 consensus_queue.cc:237] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } }
I20240502 01:58:03.367889 19850 catalog_manager.cc:5484] T be7c73395b8349d6b5256ba29154af73 P b5c209d86d3a4f459d8b8f3ee5cab66f reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c209d86d3a4f459d8b8f3ee5cab66f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c209d86d3a4f459d8b8f3ee5cab66f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 43885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6c8ed6ab3746269ba7cdd5a4874579" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe0c2501f074a368b609acea2e52b06" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42643 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:03.400041 18380 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:58:03.401227 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b5c209d86d3a4f459d8b8f3ee5cab66f to finish bootstrapping
I20240502 01:58:03.405655 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ed66cf01c18543fe9305f45ddd947d34 to finish bootstrapping
I20240502 01:58:03.410712 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e40a8be211714d24b0df48a4137bffb6 to finish bootstrapping
I20240502 01:58:03.415799 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bbe0c2501f074a368b609acea2e52b06 to finish bootstrapping
I20240502 01:58:03.419878 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bb6c8ed6ab3746269ba7cdd5a4874579 to finish bootstrapping
I20240502 01:58:03.423746 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 19913
W20240502 01:58:03.470405 20566 tablet.cc:2369] T be7c73395b8349d6b5256ba29154af73 P bb6c8ed6ab3746269ba7cdd5a4874579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:03.593502 20237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.593519 20499 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.597442 20106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.607198 20368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.666653 20368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.667071 20499 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.667349 20237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.670102 20106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.734828 20172 heartbeater.cc:503] Master 127.17.243.62:33101 was elected leader, sending a full tablet report...
I20240502 01:58:03.752421 20237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.753352 20499 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.753662 20368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.756043 20106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.907138 20303 heartbeater.cc:503] Master 127.17.243.62:33101 was elected leader, sending a full tablet report...
I20240502 01:58:03.950171 20106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.950358 20368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.950673 20499 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.950924 20237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.015831 20499 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.016438 20368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.022239 20237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.027014 20106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.107554 20499 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.108278 20237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.109593 20368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.118021 20106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.146448 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 20044
I20240502 01:58:04.151379 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 20175
I20240502 01:58:04.156256 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 20306
I20240502 01:58:04.160594 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 20437
I20240502 01:58:04.164873 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 19822
2024-05-02T01:58:04Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2012 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2012 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240502 01:58:04.184937 18380 test_util.cc:268] Using random seed: -284887706
I20240502 01:58:04.185616 18380 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:58:04.185652 18380 ts_itest-base.cc:113] --------------
I20240502 01:58:04.185665 18380 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:58:04.185678 18380 ts_itest-base.cc:115] 1 replicas per TS
I20240502 01:58:04.185690 18380 ts_itest-base.cc:116] --------------
2024-05-02T01:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:58:04Z Disabled control of system clock
I20240502 01:58:04.192620 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:43303
--webserver_interface=127.17.243.62
--webserver_port=0
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.243.62:43303
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:58:04.291564 20780 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:58:04.291765 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.291800 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:04.291859 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.294775 20780 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:04.294838 20780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:58:04.294869 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:04.294901 20780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:58:04.294925 20780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:58:04.298359 20780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.243.62:43303
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.243.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:04.299065 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:04.300012 20780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:04.306453 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.306838 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.307324 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:04.307792 20780 server_base.cc:1034] running on GCE node
I20240502 01:58:04.308167 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:04.308902 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:04.310083 20780 hybrid_clock.cc:648] HybridClock initialized: now 1714615084310073 us; error 27 us; skew 500 ppm
I20240502 01:58:04.312353 20780 webserver.cc:460] Webserver started at http://127.17.243.62:45257/ using document root <none> and password file <none>
I20240502 01:58:04.312878 20780 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:04.312944 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:04.313131 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:04.314702 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "93e5e9f48cd5489d9256ca43be967240"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.315146 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "93e5e9f48cd5489d9256ca43be967240"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.318552 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:58:04.320659 20794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.321615 20780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:04.321727 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
uuid: "93e5e9f48cd5489d9256ca43be967240"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.321815 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:04.339835 20780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:04.340384 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:04.340529 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:04.347144 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.62:43303 every 8 connection(s)
I20240502 01:58:04.347142 20780 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.62:43303
I20240502 01:58:04.348032 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:58:04.350253 20847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:04.354652 20847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240: Bootstrap starting.
I20240502 01:58:04.356536 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 20780
I20240502 01:58:04.356592 20847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:04.356662 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:58:04.357184 20847 log.cc:824] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:04.359138 20847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240: No bootstrap required, opened a new log
I20240502 01:58:04.361353 20847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e5e9f48cd5489d9256ca43be967240" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 43303 } }
I20240502 01:58:04.361513 20847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:04.361567 20847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e5e9f48cd5489d9256ca43be967240, State: Initialized, Role: FOLLOWER
I20240502 01:58:04.361984 20847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e5e9f48cd5489d9256ca43be967240" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 43303 } }
I20240502 01:58:04.362136 20847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:04.362200 20847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:04.362306 20847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:04.362939 20847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e5e9f48cd5489d9256ca43be967240" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 43303 } }
I20240502 01:58:04.363232 20847 leader_election.cc:302] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93e5e9f48cd5489d9256ca43be967240; no voters:
I20240502 01:58:04.363471 20847 leader_election.cc:288] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:04.363816 20852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:04.364048 20852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [term 1 LEADER]: Becoming Leader. State: Replica: 93e5e9f48cd5489d9256ca43be967240, State: Running, Role: LEADER
I20240502 01:58:04.364368 20852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e5e9f48cd5489d9256ca43be967240" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 43303 } }
I20240502 01:58:04.364499 20847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:04.365693 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93e5e9f48cd5489d9256ca43be967240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e5e9f48cd5489d9256ca43be967240" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 43303 } } }
I20240502 01:58:04.365801 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:04.366132 20854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e5e9f48cd5489d9256ca43be967240. Latest consensus state: current_term: 1 leader_uuid: "93e5e9f48cd5489d9256ca43be967240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e5e9f48cd5489d9256ca43be967240" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 43303 } } }
I20240502 01:58:04.366220 20862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:04.366297 20854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:04.368994 20862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:04.373428 20862 catalog_manager.cc:1347] Generated new cluster ID: 2175cdf4a50249f4af012b15c9983981
I20240502 01:58:04.373494 20862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:04.404194 20862 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:04.404969 20862 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:04.411973 20862 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 93e5e9f48cd5489d9256ca43be967240: Generated new TSK 0
I20240502 01:58:04.412540 20862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:04.415315 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.1:0
--local_ip_for_outbound_sockets=127.17.243.1
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:04.521296 20871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.521473 20871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:04.521514 20871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.524397 20871 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:04.524454 20871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:04.524521 20871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.1
I20240502 01:58:04.528012 20871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.20871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.1
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:04.528708 20871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:04.529651 20871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:04.536969 20879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.537240 20877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.537766 20876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:04.538344 20871 server_base.cc:1034] running on GCE node
I20240502 01:58:04.538640 20871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:04.539099 20871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:04.540236 20871 hybrid_clock.cc:648] HybridClock initialized: now 1714615084540228 us; error 24 us; skew 500 ppm
I20240502 01:58:04.541682 20871 webserver.cc:460] Webserver started at http://127.17.243.1:36169/ using document root <none> and password file <none>
I20240502 01:58:04.542186 20871 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:04.542243 20871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:04.542397 20871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:04.543834 20871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3783e79423214efcba54d2bb50aa202f"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.544266 20871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3783e79423214efcba54d2bb50aa202f"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.547118 20871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 01:58:04.548970 20885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.549822 20871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:58:04.549959 20871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3783e79423214efcba54d2bb50aa202f"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.550048 20871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:04.564306 20871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:04.564824 20871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:04.564949 20871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:04.565395 20871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:04.566232 20871 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:04.566277 20871 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.566322 20871 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:04.566350 20871 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.573691 20871 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.1:46147
I20240502 01:58:04.573709 20999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.1:46147 every 8 connection(s)
I20240502 01:58:04.574445 20871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:58:04.580482 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 20871
I20240502 01:58:04.580641 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:58:04.582075 21000 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:04.582267 21000 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:04.582774 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.2:0
--local_ip_for_outbound_sockets=127.17.243.2
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:04.582804 21000 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:04.584218 20811 ts_manager.cc:194] Registered new tserver with Master: 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:04.585309 20811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.1:45265
W20240502 01:58:04.683907 21003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.684067 21003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:04.684108 21003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.687021 21003 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:04.687073 21003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:04.687160 21003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.2
I20240502 01:58:04.690506 21003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.2
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:04.691180 21003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:04.692085 21003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:04.699054 21009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.699575 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.700083 21008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:04.700616 21003 server_base.cc:1034] running on GCE node
I20240502 01:58:04.700891 21003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:04.701337 21003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:04.702481 21003 hybrid_clock.cc:648] HybridClock initialized: now 1714615084702471 us; error 22 us; skew 500 ppm
I20240502 01:58:04.703919 21003 webserver.cc:460] Webserver started at http://127.17.243.2:33889/ using document root <none> and password file <none>
I20240502 01:58:04.704370 21003 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:04.704422 21003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:04.704576 21003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:04.705996 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "580cdba303094f18bc3a325acb7095ed"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.706403 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "580cdba303094f18bc3a325acb7095ed"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.709211 21003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:04.711153 21017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.711997 21003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:04.712092 21003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "580cdba303094f18bc3a325acb7095ed"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.712163 21003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:04.737167 21003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:04.737715 21003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:04.737849 21003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:04.738328 21003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:04.739113 21003 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:04.739159 21003 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.739207 21003 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:04.739234 21003 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.746995 21003 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.2:36375
I20240502 01:58:04.747017 21130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.2:36375 every 8 connection(s)
I20240502 01:58:04.747764 21003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:58:04.748943 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21003
I20240502 01:58:04.749034 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:58:04.751165 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.3:0
--local_ip_for_outbound_sockets=127.17.243.3
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:04.756242 21131 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:04.756419 21131 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:04.756850 21131 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:04.757779 20811 ts_manager.cc:194] Registered new tserver with Master: 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:04.758525 20811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.2:42609
W20240502 01:58:04.849021 21134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.849175 21134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:04.849215 21134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.852053 21134 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:04.852103 21134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:04.852164 21134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.3
I20240502 01:58:04.855516 21134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.3
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:04.856171 21134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:04.857069 21134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:04.863930 21139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.864343 21140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.864816 21142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:04.865310 21134 server_base.cc:1034] running on GCE node
I20240502 01:58:04.865638 21134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:04.866117 21134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:04.867269 21134 hybrid_clock.cc:648] HybridClock initialized: now 1714615084867253 us; error 29 us; skew 500 ppm
I20240502 01:58:04.868734 21134 webserver.cc:460] Webserver started at http://127.17.243.3:37485/ using document root <none> and password file <none>
I20240502 01:58:04.869179 21134 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:04.869231 21134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:04.869377 21134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:04.870797 21134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8690f48d41224e379f2c6c0f25251d4b"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.871214 21134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8690f48d41224e379f2c6c0f25251d4b"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.873981 21134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:58:04.875768 21148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.876626 21134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:04.876725 21134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8690f48d41224e379f2c6c0f25251d4b"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:04.876796 21134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:04.892920 21134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:04.893396 21134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:04.893532 21134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:04.893978 21134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:04.894728 21134 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:04.894773 21134 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.894819 21134 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:04.894850 21134 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.902475 21134 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.3:35731
I20240502 01:58:04.902508 21261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.3:35731 every 8 connection(s)
I20240502 01:58:04.903209 21134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:58:04.906416 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21134
I20240502 01:58:04.906515 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:58:04.908730 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.4:0
--local_ip_for_outbound_sockets=127.17.243.4
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:04.912818 21262 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:04.913060 21262 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:04.913699 21262 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:04.914839 20811 ts_manager.cc:194] Registered new tserver with Master: 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3:35731)
I20240502 01:58:04.915423 20811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.3:33019
W20240502 01:58:05.011981 21265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.012167 21265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:05.012218 21265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.015269 21265 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:05.015327 21265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:05.015412 21265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.4
I20240502 01:58:05.018957 21265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.4
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:05.019811 21265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:05.020902 21265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:05.028898 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.029309 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.029762 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:05.030599 21265 server_base.cc:1034] running on GCE node
I20240502 01:58:05.030941 21265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:05.031445 21265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:05.032593 21265 hybrid_clock.cc:648] HybridClock initialized: now 1714615085032580 us; error 23 us; skew 500 ppm
I20240502 01:58:05.034320 21265 webserver.cc:460] Webserver started at http://127.17.243.4:36435/ using document root <none> and password file <none>
I20240502 01:58:05.034847 21265 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:05.034900 21265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:05.035076 21265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:05.036652 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "87578246c04b49188edc94b74619a22c"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.037096 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "87578246c04b49188edc94b74619a22c"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.040191 21265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 01:58:05.042157 21279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.043056 21265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:58:05.043160 21265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "87578246c04b49188edc94b74619a22c"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.043233 21265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:05.054550 21265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:05.055091 21265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:05.055228 21265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:05.055706 21265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:05.056528 21265 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:05.056574 21265 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.056623 21265 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:05.056648 21265 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.064442 21265 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.4:44049
I20240502 01:58:05.064465 21392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.4:44049 every 8 connection(s)
I20240502 01:58:05.065274 21265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:05.073181 21393 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:05.073364 21393 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:05.073843 21393 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:05.073855 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21265
I20240502 01:58:05.073971 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:58:05.074882 20811 ts_manager.cc:194] Registered new tserver with Master: 87578246c04b49188edc94b74619a22c (127.17.243.4:44049)
I20240502 01:58:05.075439 20811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.4:46785
I20240502 01:58:05.076059 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.5:0
--local_ip_for_outbound_sockets=127.17.243.5
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:05.175813 21396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.176002 21396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:05.176064 21396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.178979 21396 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:05.179044 21396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:05.179129 21396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.5
I20240502 01:58:05.182631 21396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.5
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:05.183331 21396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:05.184329 21396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:05.191783 21401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.192193 21402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.192714 21404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:05.193284 21396 server_base.cc:1034] running on GCE node
I20240502 01:58:05.193590 21396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:05.194085 21396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:05.195235 21396 hybrid_clock.cc:648] HybridClock initialized: now 1714615085195226 us; error 23 us; skew 500 ppm
I20240502 01:58:05.196826 21396 webserver.cc:460] Webserver started at http://127.17.243.5:43533/ using document root <none> and password file <none>
I20240502 01:58:05.197290 21396 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:05.197340 21396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:05.197510 21396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:05.198956 21396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "63b02225f2b444f29dcf0041105f4784"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.199383 21396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "63b02225f2b444f29dcf0041105f4784"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.202244 21396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:58:05.204121 21410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.204991 21396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:05.205081 21396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "63b02225f2b444f29dcf0041105f4784"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.205153 21396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:05.232761 21396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:05.233254 21396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:05.233379 21396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:05.233808 21396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:05.234596 21396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:05.234642 21396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.234687 21396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:05.234716 21396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.242461 21396 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.5:38921
I20240502 01:58:05.242496 21523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.5:38921 every 8 connection(s)
I20240502 01:58:05.243229 21396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:05.250641 21524 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:05.250823 21524 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:05.250943 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21396
I20240502 01:58:05.251037 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:58:05.251302 21524 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:05.252203 20810 ts_manager.cc:194] Registered new tserver with Master: 63b02225f2b444f29dcf0041105f4784 (127.17.243.5:38921)
I20240502 01:58:05.252911 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.5:60849
I20240502 01:58:05.253067 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.6:0
--local_ip_for_outbound_sockets=127.17.243.6
--webserver_interface=127.17.243.6
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:05.357530 21527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.357708 21527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:05.357758 21527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.360668 21527 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:05.360723 21527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:05.360787 21527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.6
I20240502 01:58:05.364298 21527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.243.6
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.6
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:05.365053 21527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:05.366129 21527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:05.374370 21532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.374811 21533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.375289 21535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:05.375797 21527 server_base.cc:1034] running on GCE node
I20240502 01:58:05.376128 21527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:05.376600 21527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:05.377755 21527 hybrid_clock.cc:648] HybridClock initialized: now 1714615085377741 us; error 27 us; skew 500 ppm
I20240502 01:58:05.379420 21527 webserver.cc:460] Webserver started at http://127.17.243.6:44393/ using document root <none> and password file <none>
I20240502 01:58:05.379927 21527 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:05.379999 21527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:05.380170 21527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:05.381664 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c1449418fd3344ce996bf17e7bade3b7"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.382104 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c1449418fd3344ce996bf17e7bade3b7"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.385107 21527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:05.387145 21541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.388051 21527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:05.388149 21527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c1449418fd3344ce996bf17e7bade3b7"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.388552 21527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:05.401851 21527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:05.402396 21527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:05.402544 21527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:05.402998 21527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:05.403781 21527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:05.403826 21527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.403872 21527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:05.403898 21527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.412603 21527 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.6:44947
I20240502 01:58:05.412634 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.6:44947 every 8 connection(s)
I20240502 01:58:05.413375 21527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:05.417820 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21527
I20240502 01:58:05.417913 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:58:05.419934 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.7:0
--local_ip_for_outbound_sockets=127.17.243.7
--webserver_interface=127.17.243.7
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:05.421324 21655 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:05.421540 21655 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:05.422137 21655 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:05.423117 20810 ts_manager.cc:194] Registered new tserver with Master: c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6:44947)
I20240502 01:58:05.423806 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.6:60697
W20240502 01:58:05.520718 21658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.520892 21658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:05.520936 21658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.523975 21658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:05.524035 21658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:05.524097 21658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.7
I20240502 01:58:05.527565 21658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.243.7
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.7
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:05.528223 21658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:05.529155 21658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:05.536937 21666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.537329 21663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.538007 21664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:05.538553 21658 server_base.cc:1034] running on GCE node
I20240502 01:58:05.538878 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:05.539345 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:05.540494 21658 hybrid_clock.cc:648] HybridClock initialized: now 1714615085540475 us; error 33 us; skew 500 ppm
I20240502 01:58:05.542258 21658 webserver.cc:460] Webserver started at http://127.17.243.7:39841/ using document root <none> and password file <none>
I20240502 01:58:05.542757 21658 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:05.542897 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:05.543087 21658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:05.544867 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "998441e608064fad81a173c929277899"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.545357 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "998441e608064fad81a173c929277899"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.548804 21658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20240502 01:58:05.550804 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.551710 21658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:05.551796 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "998441e608064fad81a173c929277899"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:05.551862 21658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:05.561245 21658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:05.561756 21658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:05.561877 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:05.562352 21658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:05.563148 21658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:05.563194 21658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.563230 21658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:05.563248 21658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.571404 21658 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.7:33251
I20240502 01:58:05.571473 21785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.7:33251 every 8 connection(s)
I20240502 01:58:05.572201 21658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:05.575042 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21658
I20240502 01:58:05.575139 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:58:05.580756 21786 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:05.580927 21786 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:05.581338 21786 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:05.582191 20810 ts_manager.cc:194] Registered new tserver with Master: 998441e608064fad81a173c929277899 (127.17.243.7:33251)
I20240502 01:58:05.582732 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.7:60347
I20240502 01:58:05.587446 21000 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:05.588980 18380 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:58:05.597165 20810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35128:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:58:05.612576 21065 tablet_service.cc:1467] Processing CreateTablet for tablet c13461f24fc94b938cd2e524dd8a9df2 (DEFAULT_TABLE table=TestTable [id=c2e389bfaf67455eb650d1e2df27dd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:05.613523 21065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13461f24fc94b938cd2e524dd8a9df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:05.618530 21800 tablet_bootstrap.cc:492] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:05.620211 21800 tablet_bootstrap.cc:654] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:05.620811 21800 log.cc:824] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:05.622231 21800 tablet_bootstrap.cc:492] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:05.622392 21800 ts_tablet_manager.cc:1367] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:05.624572 21800 raft_consensus.cc:348] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:05.624725 21800 raft_consensus.cc:374] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:05.624786 21800 raft_consensus.cc:729] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:05.625175 21800 consensus_queue.cc:260] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:05.625291 21800 raft_consensus.cc:388] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:05.625351 21800 raft_consensus.cc:482] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:05.625453 21800 raft_consensus.cc:3037] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:05.646553 21800 raft_consensus.cc:504] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:05.646900 21800 leader_election.cc:302] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:05.647220 21800 leader_election.cc:288] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:05.647317 21802 raft_consensus.cc:2781] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:05.647513 21802 raft_consensus.cc:686] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:05.647797 21802 consensus_queue.cc:237] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:05.647915 21131 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:05.648375 21800 ts_tablet_manager.cc:1398] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240502 01:58:05.650965 20810 catalog_manager.cc:5484] T c13461f24fc94b938cd2e524dd8a9df2 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:05.659489 18380 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:58:05.661001 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3783e79423214efcba54d2bb50aa202f to finish bootstrapping
I20240502 01:58:05.666357 18380 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 580cdba303094f18bc3a325acb7095ed to finish bootstrapping
I20240502 01:58:05.670337 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8690f48d41224e379f2c6c0f25251d4b to finish bootstrapping
I20240502 01:58:05.675236 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 87578246c04b49188edc94b74619a22c to finish bootstrapping
I20240502 01:58:05.679771 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 63b02225f2b444f29dcf0041105f4784 to finish bootstrapping
I20240502 01:58:05.684413 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c1449418fd3344ce996bf17e7bade3b7 to finish bootstrapping
I20240502 01:58:05.689263 18380 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 998441e608064fad81a173c929277899 to finish bootstrapping
I20240502 01:58:05.694329 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21134
I20240502 01:58:05.700532 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21265
I20240502 01:58:05.705073 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21396
I20240502 01:58:05.709264 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21527
I20240502 01:58:05.713589 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21658
I20240502 01:58:09.469668 20810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35128:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:09.480880 20933 tablet_service.cc:1467] Processing CreateTablet for tablet 6d31ece9e2b14c35957270ffd6a7b33f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b99bc3270af143db8e5641075113894b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.480901 20932 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbd475017a14bf8b488f2830e2bcee9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b99bc3270af143db8e5641075113894b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.480895 20931 tablet_service.cc:1467] Processing CreateTablet for tablet a8c274b327e04a85b39ef8984a6f486f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b99bc3270af143db8e5641075113894b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.481981 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d31ece9e2b14c35957270ffd6a7b33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.482293 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbd475017a14bf8b488f2830e2bcee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.482465 20931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c274b327e04a85b39ef8984a6f486f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.487139 21820 tablet_bootstrap.cc:492] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:09.488840 21820 tablet_bootstrap.cc:654] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.489670 21820 log.cc:824] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:09.491081 21820 tablet_bootstrap.cc:492] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:09.491247 21820 ts_tablet_manager.cc:1367] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:58:09.493626 21820 raft_consensus.cc:348] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.493793 21820 raft_consensus.cc:374] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.493854 21820 raft_consensus.cc:729] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.494266 21820 consensus_queue.cc:260] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.494387 21820 raft_consensus.cc:388] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.494452 21820 raft_consensus.cc:482] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.494551 21820 raft_consensus.cc:3037] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.495169 21820 raft_consensus.cc:504] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.495476 21820 leader_election.cc:302] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:09.495720 21820 leader_election.cc:288] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.495796 21822 raft_consensus.cc:2781] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.495998 21822 raft_consensus.cc:686] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:09.496297 21822 consensus_queue.cc:237] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.496479 21820 ts_tablet_manager.cc:1398] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 01:58:09.496677 21820 tablet_bootstrap.cc:492] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:09.497406 21820 tablet_bootstrap.cc:654] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.498436 21820 tablet_bootstrap.cc:492] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:09.498539 21820 ts_tablet_manager.cc:1367] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:09.498517 20810 catalog_manager.cc:5484] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.499012 21820 raft_consensus.cc:348] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.499120 21820 raft_consensus.cc:374] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.499164 21820 raft_consensus.cc:729] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.499274 21820 consensus_queue.cc:260] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.499357 21820 raft_consensus.cc:388] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.499395 21820 raft_consensus.cc:482] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.499440 21820 raft_consensus.cc:3037] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.500067 21820 raft_consensus.cc:504] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.500185 21820 leader_election.cc:302] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:09.500263 21820 leader_election.cc:288] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.500300 21822 raft_consensus.cc:2781] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.500366 21822 raft_consensus.cc:686] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:09.500455 21820 ts_tablet_manager.cc:1398] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:58:09.500458 21822 consensus_queue.cc:237] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.500583 21820 tablet_bootstrap.cc:492] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:09.501289 21820 tablet_bootstrap.cc:654] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.501837 20810 catalog_manager.cc:5484] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.502244 21820 tablet_bootstrap.cc:492] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:09.502296 21820 ts_tablet_manager.cc:1367] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:09.502775 21820 raft_consensus.cc:348] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.502853 21820 raft_consensus.cc:374] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.502873 21820 raft_consensus.cc:729] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.502972 21820 consensus_queue.cc:260] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.503033 21820 raft_consensus.cc:388] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.503060 21820 raft_consensus.cc:482] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.503121 21820 raft_consensus.cc:3037] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.503731 21820 raft_consensus.cc:504] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.503832 21820 leader_election.cc:302] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:09.503916 21820 leader_election.cc:288] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.504047 21822 raft_consensus.cc:2781] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.504132 21822 raft_consensus.cc:686] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:09.504149 21820 ts_tablet_manager.cc:1398] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:09.504233 21822 consensus_queue.cc:237] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.505365 20810 catalog_manager.cc:5484] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:09.619247 21831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:09.619436 21831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:09.622524 21831 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 74.8672 ms
time per row: 0.0374336 ms
I20240502 01:58:09.739460 20811 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35128:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:09.744448 21065 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c72f9e6e34cae985bce2cab15cfd4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5508a27a4396491786f2f6fd3b5a8632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.744454 21064 tablet_service.cc:1467] Processing CreateTablet for tablet 194f70e09ac6452fbc18ef8481334607 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5508a27a4396491786f2f6fd3b5a8632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.744450 21063 tablet_service.cc:1467] Processing CreateTablet for tablet c88b79b757d8428a89debee2187ed755 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5508a27a4396491786f2f6fd3b5a8632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.744776 21065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c72f9e6e34cae985bce2cab15cfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.744930 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194f70e09ac6452fbc18ef8481334607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.745088 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88b79b757d8428a89debee2187ed755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.746876 21863 tablet_bootstrap.cc:492] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:09.747812 21863 tablet_bootstrap.cc:654] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.748742 21863 tablet_bootstrap.cc:492] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:09.748816 21863 ts_tablet_manager.cc:1367] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:09.749277 21863 raft_consensus.cc:348] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.749366 21863 raft_consensus.cc:374] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.749393 21863 raft_consensus.cc:729] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.749504 21863 consensus_queue.cc:260] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.749579 21863 raft_consensus.cc:388] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.749615 21863 raft_consensus.cc:482] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.749660 21863 raft_consensus.cc:3037] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.750401 21863 raft_consensus.cc:504] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.750524 21863 leader_election.cc:302] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:09.750680 21863 leader_election.cc:288] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.750773 21865 raft_consensus.cc:2781] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.750916 21863 ts_tablet_manager.cc:1398] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:09.751086 21863 tablet_bootstrap.cc:492] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:09.751178 21865 raft_consensus.cc:686] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:09.751307 21865 consensus_queue.cc:237] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.751876 21863 tablet_bootstrap.cc:654] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.752689 20811 catalog_manager.cc:5484] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.753047 21863 tablet_bootstrap.cc:492] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:09.753118 21863 ts_tablet_manager.cc:1367] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:09.753522 21863 raft_consensus.cc:348] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.753599 21863 raft_consensus.cc:374] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.753640 21863 raft_consensus.cc:729] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.753743 21863 consensus_queue.cc:260] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.753810 21863 raft_consensus.cc:388] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.753854 21863 raft_consensus.cc:482] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.753891 21863 raft_consensus.cc:3037] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.754576 21863 raft_consensus.cc:504] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.754694 21863 leader_election.cc:302] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:09.754781 21863 leader_election.cc:288] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.754833 21865 raft_consensus.cc:2781] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.754925 21865 raft_consensus.cc:686] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:09.754957 21863 ts_tablet_manager.cc:1398] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:09.755040 21865 consensus_queue.cc:237] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.755098 21863 tablet_bootstrap.cc:492] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:09.755848 21863 tablet_bootstrap.cc:654] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.756388 20811 catalog_manager.cc:5484] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.756769 21863 tablet_bootstrap.cc:492] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:09.756847 21863 ts_tablet_manager.cc:1367] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:09.757238 21863 raft_consensus.cc:348] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.757326 21863 raft_consensus.cc:374] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.757364 21863 raft_consensus.cc:729] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.757462 21863 consensus_queue.cc:260] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.757534 21863 raft_consensus.cc:388] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.757570 21863 raft_consensus.cc:482] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.757616 21863 raft_consensus.cc:3037] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.758244 21863 raft_consensus.cc:504] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.758337 21863 leader_election.cc:302] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:09.758401 21863 leader_election.cc:288] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.758481 21867 raft_consensus.cc:2781] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.758558 21867 raft_consensus.cc:686] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:09.758589 21863 ts_tablet_manager.cc:1398] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:09.758651 21867 consensus_queue.cc:237] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:09.759688 20811 catalog_manager.cc:5484] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:09.859474 21874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:09.859660 21874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:09.862550 21874 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 63.9613 ms
time per row: 0.0319807 ms
I20240502 01:58:09.961990 20810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35128:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:09.966997 20932 tablet_service.cc:1467] Processing CreateTablet for tablet 3d024eee194347779dd13f423cc9023c (DEFAULT_TABLE table=rebalance_test_table_2 [id=e5eb5c4bfbb140ec9d8781f1ca346fab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.967314 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d024eee194347779dd13f423cc9023c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.968919 21820 tablet_bootstrap.cc:492] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:09.969098 20933 tablet_service.cc:1467] Processing CreateTablet for tablet 14f887acbed24c2fac257df46bcc3a8c (DEFAULT_TABLE table=rebalance_test_table_2 [id=e5eb5c4bfbb140ec9d8781f1ca346fab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.969326 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f887acbed24c2fac257df46bcc3a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.969681 21820 tablet_bootstrap.cc:654] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.970641 21820 tablet_bootstrap.cc:492] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:09.970716 21820 ts_tablet_manager.cc:1367] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:09.966997 20931 tablet_service.cc:1467] Processing CreateTablet for tablet c4fd077b7bcc43478ba93f21de746397 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e5eb5c4bfbb140ec9d8781f1ca346fab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:09.970989 20931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fd077b7bcc43478ba93f21de746397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:09.971055 21820 raft_consensus.cc:348] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.971156 21820 raft_consensus.cc:374] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.971196 21820 raft_consensus.cc:729] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.971307 21820 consensus_queue.cc:260] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.971370 21820 raft_consensus.cc:388] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.971403 21820 raft_consensus.cc:482] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.971457 21820 raft_consensus.cc:3037] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.972040 21820 raft_consensus.cc:504] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.972148 21820 leader_election.cc:302] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:09.972232 21820 leader_election.cc:288] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.972280 21822 raft_consensus.cc:2781] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.972352 21822 raft_consensus.cc:686] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:09.972476 21820 ts_tablet_manager.cc:1398] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:09.972446 21822 consensus_queue.cc:237] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.972622 21820 tablet_bootstrap.cc:492] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:09.973536 21820 tablet_bootstrap.cc:654] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.974157 20809 catalog_manager.cc:5484] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.974431 21820 tablet_bootstrap.cc:492] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:09.974493 21820 ts_tablet_manager.cc:1367] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:09.974797 21820 raft_consensus.cc:348] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.974866 21820 raft_consensus.cc:374] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.974885 21820 raft_consensus.cc:729] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.974979 21820 consensus_queue.cc:260] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.975039 21820 raft_consensus.cc:388] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.975059 21820 raft_consensus.cc:482] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.975100 21820 raft_consensus.cc:3037] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.975817 21820 raft_consensus.cc:504] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.975939 21820 leader_election.cc:302] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:09.976063 21822 raft_consensus.cc:2781] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.976133 21822 raft_consensus.cc:686] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:09.976234 21822 consensus_queue.cc:237] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.976521 21820 leader_election.cc:288] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.976739 21820 ts_tablet_manager.cc:1398] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:09.976876 21820 tablet_bootstrap.cc:492] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:09.977499 20809 catalog_manager.cc:5484] T 14f887acbed24c2fac257df46bcc3a8c P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:09.977816 21820 tablet_bootstrap.cc:654] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:09.978709 21820 tablet_bootstrap.cc:492] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:09.978788 21820 ts_tablet_manager.cc:1367] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:09.979131 21820 raft_consensus.cc:348] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.979203 21820 raft_consensus.cc:374] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:09.979223 21820 raft_consensus.cc:729] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:09.979317 21820 consensus_queue.cc:260] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.979394 21820 raft_consensus.cc:388] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:09.979436 21820 raft_consensus.cc:482] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:09.979488 21820 raft_consensus.cc:3037] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:09.980082 21820 raft_consensus.cc:504] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.980177 21820 leader_election.cc:302] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:09.980276 21820 leader_election.cc:288] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:09.980324 21822 raft_consensus.cc:2781] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:09.980419 21822 raft_consensus.cc:686] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:09.980525 21822 consensus_queue.cc:237] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:09.980526 21820 ts_tablet_manager.cc:1398] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:09.982640 20809 catalog_manager.cc:5484] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:10.082247 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:10.082433 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:10.085328 21913 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 81.068 ms
time per row: 0.040534 ms
I20240502 01:58:10.201900 20809 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35128:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:10.206699 21063 tablet_service.cc:1467] Processing CreateTablet for tablet 79511d04d45d4c00821f7bcc3231c1a9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43fc2eb6b2854abb8a75cfbccd8844a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:10.206785 21065 tablet_service.cc:1467] Processing CreateTablet for tablet cad36bda76a24e3da0bcd937b30d48a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43fc2eb6b2854abb8a75cfbccd8844a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:10.206785 21064 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd8d25f42446dc8c832b9ca46f17c8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=43fc2eb6b2854abb8a75cfbccd8844a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:10.207012 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79511d04d45d4c00821f7bcc3231c1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:10.207201 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd8d25f42446dc8c832b9ca46f17c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:10.207343 21065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad36bda76a24e3da0bcd937b30d48a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:10.208775 21863 tablet_bootstrap.cc:492] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:10.209544 21863 tablet_bootstrap.cc:654] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:10.210464 21863 tablet_bootstrap.cc:492] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:10.210543 21863 ts_tablet_manager.cc:1367] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:10.210929 21863 raft_consensus.cc:348] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.211009 21863 raft_consensus.cc:374] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:10.211045 21863 raft_consensus.cc:729] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:10.211169 21863 consensus_queue.cc:260] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.211232 21863 raft_consensus.cc:388] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:10.211264 21863 raft_consensus.cc:482] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:10.211308 21863 raft_consensus.cc:3037] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:10.211866 21863 raft_consensus.cc:504] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.212005 21863 leader_election.cc:302] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:10.212117 21863 leader_election.cc:288] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:10.212175 21867 raft_consensus.cc:2781] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:10.212287 21867 raft_consensus.cc:686] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:10.212401 21867 consensus_queue.cc:237] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.212440 21863 ts_tablet_manager.cc:1398] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:10.212627 21863 tablet_bootstrap.cc:492] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:10.213452 21863 tablet_bootstrap.cc:654] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:10.213613 20809 catalog_manager.cc:5484] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:10.214375 21863 tablet_bootstrap.cc:492] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:10.214473 21863 ts_tablet_manager.cc:1367] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:10.214825 21863 raft_consensus.cc:348] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.214903 21863 raft_consensus.cc:374] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:10.214932 21863 raft_consensus.cc:729] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:10.215688 21863 consensus_queue.cc:260] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.215893 21863 raft_consensus.cc:388] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:10.215937 21863 raft_consensus.cc:482] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:10.215967 21863 raft_consensus.cc:3037] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:10.216732 21863 raft_consensus.cc:504] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.216838 21863 leader_election.cc:302] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:10.216928 21863 leader_election.cc:288] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:10.216964 21867 raft_consensus.cc:2781] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:10.217029 21867 raft_consensus.cc:686] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:10.217123 21867 consensus_queue.cc:237] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.217131 21863 ts_tablet_manager.cc:1398] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:10.217559 21863 tablet_bootstrap.cc:492] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Bootstrap starting.
I20240502 01:58:10.218263 20809 catalog_manager.cc:5484] T 92bd8d25f42446dc8c832b9ca46f17c8 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:10.218361 21863 tablet_bootstrap.cc:654] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:10.219775 21863 tablet_bootstrap.cc:492] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: No bootstrap required, opened a new log
I20240502 01:58:10.219858 21863 ts_tablet_manager.cc:1367] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:10.220191 21863 raft_consensus.cc:348] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.220264 21863 raft_consensus.cc:374] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:10.220304 21863 raft_consensus.cc:729] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Initialized, Role: FOLLOWER
I20240502 01:58:10.220420 21863 consensus_queue.cc:260] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.220510 21863 raft_consensus.cc:388] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:10.220546 21863 raft_consensus.cc:482] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:10.220594 21863 raft_consensus.cc:3037] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:10.221166 21863 raft_consensus.cc:504] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.221266 21863 leader_election.cc:302] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed; no voters:
I20240502 01:58:10.221349 21863 leader_election.cc:288] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:10.221383 21867 raft_consensus.cc:2781] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:10.221469 21867 raft_consensus.cc:686] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Leader. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:10.221560 21867 consensus_queue.cc:237] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } }
I20240502 01:58:10.221587 21863 ts_tablet_manager.cc:1398] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:10.222605 20809 catalog_manager.cc:5484] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 580cdba303094f18bc3a325acb7095ed (127.17.243.2). New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:10.323153 21943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:10.323372 21943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:10.326310 21943 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 53.2418 ms
time per row: 0.0266209 ms
I20240502 01:58:10.416690 20810 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35128:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:10.421411 20931 tablet_service.cc:1467] Processing CreateTablet for tablet 56b2393c70cb4dcfbc7e8ba06e67124b (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b1ddb13c1824db89bc83e60ee62397f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:10.421411 20932 tablet_service.cc:1467] Processing CreateTablet for tablet b164633a362e4318975b74a1c5ea362e (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b1ddb13c1824db89bc83e60ee62397f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:10.421443 20933 tablet_service.cc:1467] Processing CreateTablet for tablet 795c3440189a488ea97073a9f2e5856f (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b1ddb13c1824db89bc83e60ee62397f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:10.421761 20931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b2393c70cb4dcfbc7e8ba06e67124b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:10.421922 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795c3440189a488ea97073a9f2e5856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:10.422044 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b164633a362e4318975b74a1c5ea362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:10.423480 21820 tablet_bootstrap.cc:492] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:10.424173 21820 tablet_bootstrap.cc:654] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:10.425083 21820 tablet_bootstrap.cc:492] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:10.425160 21820 ts_tablet_manager.cc:1367] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:10.425638 21820 raft_consensus.cc:348] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.425719 21820 raft_consensus.cc:374] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:10.425746 21820 raft_consensus.cc:729] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:10.425884 21820 consensus_queue.cc:260] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.425997 21820 raft_consensus.cc:388] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:10.426040 21820 raft_consensus.cc:482] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:10.426091 21820 raft_consensus.cc:3037] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:10.426697 21820 raft_consensus.cc:504] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.426807 21820 leader_election.cc:302] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:10.426896 21820 leader_election.cc:288] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:10.426986 21823 raft_consensus.cc:2781] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:10.427081 21823 raft_consensus.cc:686] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:10.427115 21820 ts_tablet_manager.cc:1398] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:10.427232 21820 tablet_bootstrap.cc:492] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:10.427183 21823 consensus_queue.cc:237] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.428232 21820 tablet_bootstrap.cc:654] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:10.428485 20810 catalog_manager.cc:5484] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:10.429224 21820 tablet_bootstrap.cc:492] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:10.429308 21820 ts_tablet_manager.cc:1367] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:10.429709 21820 raft_consensus.cc:348] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.429801 21820 raft_consensus.cc:374] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:10.429839 21820 raft_consensus.cc:729] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:10.429981 21820 consensus_queue.cc:260] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.430073 21820 raft_consensus.cc:388] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:10.430112 21820 raft_consensus.cc:482] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:10.430156 21820 raft_consensus.cc:3037] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:10.430806 21820 raft_consensus.cc:504] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.430920 21820 leader_election.cc:302] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:10.431002 21820 leader_election.cc:288] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:10.431056 21823 raft_consensus.cc:2781] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:10.431159 21823 raft_consensus.cc:686] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:10.431227 21820 ts_tablet_manager.cc:1398] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:10.431268 21823 consensus_queue.cc:237] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.431370 21820 tablet_bootstrap.cc:492] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Bootstrap starting.
I20240502 01:58:10.432149 21820 tablet_bootstrap.cc:654] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:10.432783 20810 catalog_manager.cc:5484] T b164633a362e4318975b74a1c5ea362e P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:10.433696 21820 tablet_bootstrap.cc:492] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: No bootstrap required, opened a new log
I20240502 01:58:10.433795 21820 ts_tablet_manager.cc:1367] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:10.434278 21820 raft_consensus.cc:348] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.434417 21820 raft_consensus.cc:374] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:10.434464 21820 raft_consensus.cc:729] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Initialized, Role: FOLLOWER
I20240502 01:58:10.434595 21820 consensus_queue.cc:260] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.434679 21820 raft_consensus.cc:388] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:10.434710 21820 raft_consensus.cc:482] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:10.434747 21820 raft_consensus.cc:3037] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:10.435310 21820 raft_consensus.cc:504] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.435431 21820 leader_election.cc:302] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f; no voters:
I20240502 01:58:10.435503 21820 leader_election.cc:288] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:10.435554 21823 raft_consensus.cc:2781] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:10.435645 21823 raft_consensus.cc:686] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Leader. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:10.435714 21820 ts_tablet_manager.cc:1398] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:10.435771 21823 consensus_queue.cc:237] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } }
I20240502 01:58:10.437079 20810 catalog_manager.cc:5484] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783e79423214efcba54d2bb50aa202f (127.17.243.1). New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:10.541980 21973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:10.542168 21973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:10.545020 21973 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 71.1364 ms
time per row: 0.0355682 ms
I20240502 01:58:10.651448 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.3:35731
--local_ip_for_outbound_sockets=127.17.243.3
--tserver_master_addrs=127.17.243.62:43303
--webserver_port=37485
--webserver_interface=127.17.243.3
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:10.751982 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:10.752192 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:10.752252 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:10.756296 21997 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:10.756373 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:10.756476 21997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.3
I20240502 01:58:10.761226 21997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.3:35731
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.243.3
--webserver_port=37485
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.21997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.3
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:10.762076 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:10.763222 21997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:10.771716 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:10.772017 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:10.772569 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:10.773015 21997 server_base.cc:1034] running on GCE node
I20240502 01:58:10.773329 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:10.773815 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:10.774963 21997 hybrid_clock.cc:648] HybridClock initialized: now 1714615090774946 us; error 30 us; skew 500 ppm
I20240502 01:58:10.776712 21997 webserver.cc:460] Webserver started at http://127.17.243.3:37485/ using document root <none> and password file <none>
I20240502 01:58:10.777278 21997 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:10.777357 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:10.780596 21997 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:10.782313 22011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:10.783290 21997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:10.783402 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8690f48d41224e379f2c6c0f25251d4b"
format_stamp: "Formatted at 2024-05-02 01:58:04 on dist-test-slave-j3s5"
I20240502 01:58:10.783762 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:10.807883 21997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:10.808370 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:10.808494 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:10.808923 21997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:10.809684 21997 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:10.809728 21997 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:10.809762 21997 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:10.809782 21997 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:10.818256 21997 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.3:35731
I20240502 01:58:10.818275 22124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.3:35731 every 8 connection(s)
I20240502 01:58:10.819183 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:58:10.826588 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 21997
I20240502 01:58:10.827011 22125 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:10.827025 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.4:44049
--local_ip_for_outbound_sockets=127.17.243.4
--tserver_master_addrs=127.17.243.62:43303
--webserver_port=36435
--webserver_interface=127.17.243.4
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:10.827203 22125 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:10.827637 22125 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:10.828708 20810 ts_manager.cc:194] Re-registered known tserver with Master: 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3:35731)
I20240502 01:58:10.829367 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.3:46471
W20240502 01:58:10.925627 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:10.925793 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:10.925849 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:10.928715 22128 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:10.928774 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:10.928850 22128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.4
I20240502 01:58:10.932238 22128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.4:44049
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.243.4
--webserver_port=36435
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.22128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.4
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:10.932914 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:10.933835 22128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:10.940994 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:10.941298 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:10.941907 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:10.942368 22128 server_base.cc:1034] running on GCE node
I20240502 01:58:10.942692 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:10.943184 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:10.944344 22128 hybrid_clock.cc:648] HybridClock initialized: now 1714615090944323 us; error 34 us; skew 500 ppm
I20240502 01:58:10.946020 22128 webserver.cc:460] Webserver started at http://127.17.243.4:36435/ using document root <none> and password file <none>
I20240502 01:58:10.946508 22128 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:10.946586 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:10.949602 22128 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:10.951424 22142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:10.952359 22128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:10.952463 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "87578246c04b49188edc94b74619a22c"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:10.952857 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:10.968894 22128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:10.969377 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:10.969514 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:10.969981 22128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:10.970739 22128 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:10.970785 22128 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:10.970832 22128 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:10.970860 22128 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:10.978683 22128 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.4:44049
I20240502 01:58:10.978734 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.4:44049 every 8 connection(s)
I20240502 01:58:10.979529 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:10.982368 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 22128
I20240502 01:58:10.982705 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.5:38921
--local_ip_for_outbound_sockets=127.17.243.5
--tserver_master_addrs=127.17.243.62:43303
--webserver_port=43533
--webserver_interface=127.17.243.5
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:10.987036 22256 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:10.987238 22256 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:10.987706 22256 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:10.988656 20810 ts_manager.cc:194] Re-registered known tserver with Master: 87578246c04b49188edc94b74619a22c (127.17.243.4:44049)
I20240502 01:58:10.989382 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.4:52101
W20240502 01:58:11.081224 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:11.081383 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:11.081449 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:11.084314 22259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:11.084372 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:11.084456 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.5
I20240502 01:58:11.087991 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.5:38921
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.243.5
--webserver_port=43533
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.5
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:11.088663 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:11.089572 22259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:11.096678 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:11.097003 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:11.097512 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:11.097985 22259 server_base.cc:1034] running on GCE node
I20240502 01:58:11.098265 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:11.098691 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:11.099831 22259 hybrid_clock.cc:648] HybridClock initialized: now 1714615091099811 us; error 33 us; skew 500 ppm
I20240502 01:58:11.101356 22259 webserver.cc:460] Webserver started at http://127.17.243.5:43533/ using document root <none> and password file <none>
I20240502 01:58:11.101816 22259 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:11.101884 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:11.104756 22259 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:11.106389 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.107231 22259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:11.107331 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "63b02225f2b444f29dcf0041105f4784"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:11.107666 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:11.137039 22259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:11.137537 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:11.137671 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:11.138142 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:11.138990 22259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:11.139037 22259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.139082 22259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:11.139109 22259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.146709 22259 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.5:38921
I20240502 01:58:11.146735 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.5:38921 every 8 connection(s)
I20240502 01:58:11.147564 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:11.155009 22387 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:11.155181 22387 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:11.155632 22387 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:11.156484 20810 ts_manager.cc:194] Re-registered known tserver with Master: 63b02225f2b444f29dcf0041105f4784 (127.17.243.5:38921)
I20240502 01:58:11.157001 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.5:51713
I20240502 01:58:11.157691 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 22259
I20240502 01:58:11.158113 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.6:44947
--local_ip_for_outbound_sockets=127.17.243.6
--tserver_master_addrs=127.17.243.62:43303
--webserver_port=44393
--webserver_interface=127.17.243.6
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:11.255447 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:11.255602 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:11.255645 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:11.258653 22390 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:11.258708 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:11.258795 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.6
I20240502 01:58:11.262148 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.6:44947
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.243.6
--webserver_port=44393
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.22390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.6
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:11.262789 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:11.263664 22390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:11.270555 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:11.271037 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:11.271462 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:11.271917 22390 server_base.cc:1034] running on GCE node
I20240502 01:58:11.272218 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:11.272667 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:11.273799 22390 hybrid_clock.cc:648] HybridClock initialized: now 1714615091273786 us; error 30 us; skew 500 ppm
I20240502 01:58:11.275235 22390 webserver.cc:460] Webserver started at http://127.17.243.6:44393/ using document root <none> and password file <none>
I20240502 01:58:11.275691 22390 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:11.275758 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:11.278620 22390 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:11.280177 22404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.281005 22390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:11.281096 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c1449418fd3344ce996bf17e7bade3b7"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:11.281419 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:11.300923 22390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:11.301401 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:11.301543 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:11.302004 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:11.302747 22390 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:11.302793 22390 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.302839 22390 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:11.302865 22390 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.311004 22390 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.6:44947
I20240502 01:58:11.311043 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.6:44947 every 8 connection(s)
I20240502 01:58:11.311851 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:11.312572 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 22390
I20240502 01:58:11.312922 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.7:33251
--local_ip_for_outbound_sockets=127.17.243.7
--tserver_master_addrs=127.17.243.62:43303
--webserver_port=39841
--webserver_interface=127.17.243.7
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:11.320382 22518 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:11.320588 22518 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:11.321040 22518 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:11.321903 20810 ts_manager.cc:194] Re-registered known tserver with Master: c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6:44947)
I20240502 01:58:11.322448 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.6:47775
W20240502 01:58:11.416469 22521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:11.416635 22521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:11.416688 22521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:11.419567 22521 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:11.419623 22521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:11.419700 22521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.7
I20240502 01:58:11.423106 22521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.7:33251
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.243.7
--webserver_port=39841
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.22521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.7
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:11.423784 22521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:11.424723 22521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:11.431794 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:11.432231 22529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:11.432610 22526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:11.433027 22521 server_base.cc:1034] running on GCE node
I20240502 01:58:11.433332 22521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:11.433769 22521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:11.434914 22521 hybrid_clock.cc:648] HybridClock initialized: now 1714615091434901 us; error 24 us; skew 500 ppm
I20240502 01:58:11.436378 22521 webserver.cc:460] Webserver started at http://127.17.243.7:39841/ using document root <none> and password file <none>
I20240502 01:58:11.436838 22521 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:11.436906 22521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:11.439791 22521 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:11.441572 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.442487 22521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:11.442595 22521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "998441e608064fad81a173c929277899"
format_stamp: "Formatted at 2024-05-02 01:58:05 on dist-test-slave-j3s5"
I20240502 01:58:11.442950 22521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:11.456218 22521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:11.456693 22521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:11.456804 22521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:11.457199 22521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:11.457978 22521 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:11.458021 22521 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.458055 22521 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:11.458074 22521 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:11.465538 22521 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.7:33251
I20240502 01:58:11.465561 22649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.7:33251 every 8 connection(s)
I20240502 01:58:11.466382 22521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:11.468250 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 22521
I20240502 01:58:11.473781 22650 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:11.474011 22650 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:11.474575 22650 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:11.475492 20810 ts_manager.cc:194] Re-registered known tserver with Master: 998441e608064fad81a173c929277899 (127.17.243.7:33251)
I20240502 01:58:11.475982 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.7:33933
I20240502 01:58:11.646335 20933 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:58:11.647994 21063 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:11.651204 22190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.652042 22584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.654120 22059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.654788 22452 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.662403 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.761325 20933 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:58:11.761325 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.765913 22059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.773743 22584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.774435 22190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.785100 21063 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:11.795156 22452 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.820649 21085 consensus_queue.cc:237] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } }
I20240502 01:58:11.822108 22740 raft_consensus.cc:2932] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 998441e608064fad81a173c929277899 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } } }
I20240502 01:58:11.824319 20808 catalog_manager.cc:5484] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index -1 to 4, NON_VOTER 998441e608064fad81a173c929277899 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.830269 21020 consensus_peers.cc:485] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed -> Peer 998441e608064fad81a173c929277899 (127.17.243.7:33251): Couldn't send request to peer 998441e608064fad81a173c929277899. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a5c72f9e6e34cae985bce2cab15cfd4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.831208 22125 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:11.833218 20953 consensus_queue.cc:237] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } }
I20240502 01:58:11.834563 22754 raft_consensus.cc:2932] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } } }
I20240502 01:58:11.836761 20810 catalog_manager.cc:5484] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.838768 20888 consensus_peers.cc:485] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f -> Peer 63b02225f2b444f29dcf0041105f4784 (127.17.243.5:38921): Couldn't send request to peer 63b02225f2b444f29dcf0041105f4784. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8c274b327e04a85b39ef8984a6f486f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.842381 21085 consensus_queue.cc:237] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } }
I20240502 01:58:11.843142 22740 raft_consensus.cc:2932] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } } }
I20240502 01:58:11.845175 20810 catalog_manager.cc:5484] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index -1 to 4, NON_VOTER c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.847306 21018 consensus_peers.cc:485] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed -> Peer c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6:44947): Couldn't send request to peer c1449418fd3344ce996bf17e7bade3b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cad36bda76a24e3da0bcd937b30d48a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.851400 20953 consensus_queue.cc:237] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } }
I20240502 01:58:11.852190 22754 raft_consensus.cc:2932] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 87578246c04b49188edc94b74619a22c (127.17.243.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } } }
I20240502 01:58:11.854068 20810 catalog_manager.cc:5484] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER 87578246c04b49188edc94b74619a22c (127.17.243.4) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.855931 20888 consensus_peers.cc:485] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f -> Peer 87578246c04b49188edc94b74619a22c (127.17.243.4:44049): Couldn't send request to peer 87578246c04b49188edc94b74619a22c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 795c3440189a488ea97073a9f2e5856f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.860167 20953 consensus_queue.cc:237] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } }
I20240502 01:58:11.861012 22753 raft_consensus.cc:2932] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } } }
I20240502 01:58:11.863512 20810 catalog_manager.cc:5484] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.864558 20889 consensus_peers.cc:485] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f -> Peer 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3:35731): Couldn't send request to peer 8690f48d41224e379f2c6c0f25251d4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56b2393c70cb4dcfbc7e8ba06e67124b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.868597 20953 consensus_queue.cc:237] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } }
I20240502 01:58:11.869359 22753 raft_consensus.cc:2932] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 998441e608064fad81a173c929277899 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } } }
I20240502 01:58:11.871073 20810 catalog_manager.cc:5484] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER 998441e608064fad81a173c929277899 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.873075 20888 consensus_peers.cc:485] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f -> Peer 998441e608064fad81a173c929277899 (127.17.243.7:33251): Couldn't send request to peer 998441e608064fad81a173c929277899. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fbd475017a14bf8b488f2830e2bcee9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.877007 21085 consensus_queue.cc:237] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } }
I20240502 01:58:11.877748 22740 raft_consensus.cc:2932] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } } }
I20240502 01:58:11.879643 20810 catalog_manager.cc:5484] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index -1 to 4, NON_VOTER 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.881404 21020 consensus_peers.cc:485] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed -> Peer 63b02225f2b444f29dcf0041105f4784 (127.17.243.5:38921): Couldn't send request to peer 63b02225f2b444f29dcf0041105f4784. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88b79b757d8428a89debee2187ed755. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.885874 20953 consensus_queue.cc:237] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } }
I20240502 01:58:11.886770 22753 raft_consensus.cc:2932] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } } }
I20240502 01:58:11.888563 20810 catalog_manager.cc:5484] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.891278 20886 consensus_peers.cc:485] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f -> Peer c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6:44947): Couldn't send request to peer c1449418fd3344ce996bf17e7bade3b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4fd077b7bcc43478ba93f21de746397. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.896129 21085 consensus_queue.cc:237] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } }
I20240502 01:58:11.896951 22741 raft_consensus.cc:2932] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 87578246c04b49188edc94b74619a22c (127.17.243.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } } }
I20240502 01:58:11.898766 20810 catalog_manager.cc:5484] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index -1 to 4, NON_VOTER 87578246c04b49188edc94b74619a22c (127.17.243.4) added. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:11.900450 21020 consensus_peers.cc:485] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed -> Peer 87578246c04b49188edc94b74619a22c (127.17.243.4:44049): Couldn't send request to peer 87578246c04b49188edc94b74619a22c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79511d04d45d4c00821f7bcc3231c1a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:11.991148 22256 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:12.158970 22387 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:12.261804 22888 ts_tablet_manager.cc:902] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:12.262466 22888 tablet_copy_client.cc:308] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:12.266019 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet a8c274b327e04a85b39ef8984a6f486f from peer 63b02225f2b444f29dcf0041105f4784 ({username='slave'} at 127.17.243.5:46611)
I20240502 01:58:12.266141 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet a8c274b327e04a85b39ef8984a6f486f from peer 63b02225f2b444f29dcf0041105f4784 at {username='slave'} at 127.17.243.5:46611: session id = 63b02225f2b444f29dcf0041105f4784-a8c274b327e04a85b39ef8984a6f486f
I20240502 01:58:12.267313 20974 tablet_copy_source_session.cc:205] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.267419 22894 ts_tablet_manager.cc:902] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:12.268041 22894 tablet_copy_client.cc:308] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:12.270709 22888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c274b327e04a85b39ef8984a6f486f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.271567 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet c4fd077b7bcc43478ba93f21de746397 from peer c1449418fd3344ce996bf17e7bade3b7 ({username='slave'} at 127.17.243.6:45867)
I20240502 01:58:12.271659 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet c4fd077b7bcc43478ba93f21de746397 from peer c1449418fd3344ce996bf17e7bade3b7 at {username='slave'} at 127.17.243.6:45867: session id = c1449418fd3344ce996bf17e7bade3b7-c4fd077b7bcc43478ba93f21de746397
I20240502 01:58:12.272245 20974 tablet_copy_source_session.cc:205] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.274999 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fd077b7bcc43478ba93f21de746397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.276490 22888 tablet_copy_client.cc:783] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.276656 22888 tablet_copy_client.cc:647] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.278158 22888 tablet_copy_client.cc:523] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.279736 22888 tablet_bootstrap.cc:492] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Bootstrap starting.
I20240502 01:58:12.280215 22894 tablet_copy_client.cc:783] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.280375 22894 tablet_copy_client.cc:647] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.281569 22894 tablet_copy_client.cc:523] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.283120 22894 tablet_bootstrap.cc:492] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Bootstrap starting.
I20240502 01:58:12.285699 22888 log.cc:824] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:12.288890 22894 log.cc:824] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:12.312615 22888 tablet_bootstrap.cc:492] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.312904 22888 tablet_bootstrap.cc:492] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Bootstrap complete.
I20240502 01:58:12.313169 22888 ts_tablet_manager.cc:1367] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.004s
I20240502 01:58:12.315348 22888 raft_consensus.cc:348] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } }
I20240502 01:58:12.315598 22888 raft_consensus.cc:729] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63b02225f2b444f29dcf0041105f4784, State: Initialized, Role: LEARNER
I20240502 01:58:12.315907 22894 tablet_bootstrap.cc:492] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.315937 22888 consensus_queue.cc:260] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } }
I20240502 01:58:12.316171 22894 tablet_bootstrap.cc:492] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Bootstrap complete.
I20240502 01:58:12.316422 22894 ts_tablet_manager.cc:1367] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240502 01:58:12.316728 22888 ts_tablet_manager.cc:1398] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240502 01:58:12.317884 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session 63b02225f2b444f29dcf0041105f4784-a8c274b327e04a85b39ef8984a6f486f received from {username='slave'} at 127.17.243.5:46611
I20240502 01:58:12.318007 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session 63b02225f2b444f29dcf0041105f4784-a8c274b327e04a85b39ef8984a6f486f on tablet a8c274b327e04a85b39ef8984a6f486f with peer 63b02225f2b444f29dcf0041105f4784
I20240502 01:58:12.318902 22894 raft_consensus.cc:348] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } }
I20240502 01:58:12.319108 22894 raft_consensus.cc:729] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c1449418fd3344ce996bf17e7bade3b7, State: Initialized, Role: LEARNER
I20240502 01:58:12.319397 22894 consensus_queue.cc:260] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } }
I20240502 01:58:12.319958 22518 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:12.320143 22894 ts_tablet_manager.cc:1398] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 01:58:12.320727 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session c1449418fd3344ce996bf17e7bade3b7-c4fd077b7bcc43478ba93f21de746397 received from {username='slave'} at 127.17.243.6:45867
I20240502 01:58:12.320786 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session c1449418fd3344ce996bf17e7bade3b7-c4fd077b7bcc43478ba93f21de746397 on tablet c4fd077b7bcc43478ba93f21de746397 with peer c1449418fd3344ce996bf17e7bade3b7
I20240502 01:58:12.352381 22904 ts_tablet_manager.cc:902] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:12.353085 22904 tablet_copy_client.cc:308] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:12.356863 22910 ts_tablet_manager.cc:902] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:12.357244 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet 795c3440189a488ea97073a9f2e5856f from peer 87578246c04b49188edc94b74619a22c ({username='slave'} at 127.17.243.4:39485)
I20240502 01:58:12.357332 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet 795c3440189a488ea97073a9f2e5856f from peer 87578246c04b49188edc94b74619a22c at {username='slave'} at 127.17.243.4:39485: session id = 87578246c04b49188edc94b74619a22c-795c3440189a488ea97073a9f2e5856f
I20240502 01:58:12.357553 22910 tablet_copy_client.cc:308] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:12.358196 20974 tablet_copy_source_session.cc:205] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.360677 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795c3440189a488ea97073a9f2e5856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.363530 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet 56b2393c70cb4dcfbc7e8ba06e67124b from peer 8690f48d41224e379f2c6c0f25251d4b ({username='slave'} at 127.17.243.3:58795)
I20240502 01:58:12.363605 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet 56b2393c70cb4dcfbc7e8ba06e67124b from peer 8690f48d41224e379f2c6c0f25251d4b at {username='slave'} at 127.17.243.3:58795: session id = 8690f48d41224e379f2c6c0f25251d4b-56b2393c70cb4dcfbc7e8ba06e67124b
I20240502 01:58:12.364221 20974 tablet_copy_source_session.cc:205] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.365725 22904 tablet_copy_client.cc:783] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.365896 22904 tablet_copy_client.cc:647] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.366921 22910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b2393c70cb4dcfbc7e8ba06e67124b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.367231 22904 tablet_copy_client.cc:523] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.368463 22904 tablet_bootstrap.cc:492] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Bootstrap starting.
I20240502 01:58:12.371206 22910 tablet_copy_client.cc:783] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.371356 22910 tablet_copy_client.cc:647] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.372635 22910 tablet_copy_client.cc:523] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.373647 22904 log.cc:824] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:12.373889 22910 tablet_bootstrap.cc:492] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Bootstrap starting.
I20240502 01:58:12.378078 22910 log.cc:824] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:12.390501 22904 tablet_bootstrap.cc:492] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.390727 22904 tablet_bootstrap.cc:492] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Bootstrap complete.
I20240502 01:58:12.390962 22904 ts_tablet_manager.cc:1367] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.001s
I20240502 01:58:12.392666 22904 raft_consensus.cc:348] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } }
I20240502 01:58:12.392843 22904 raft_consensus.cc:729] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 87578246c04b49188edc94b74619a22c, State: Initialized, Role: LEARNER
I20240502 01:58:12.393123 22904 consensus_queue.cc:260] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } }
I20240502 01:58:12.393816 22904 ts_tablet_manager.cc:1398] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:12.394436 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session 87578246c04b49188edc94b74619a22c-795c3440189a488ea97073a9f2e5856f received from {username='slave'} at 127.17.243.4:39485
I20240502 01:58:12.394507 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session 87578246c04b49188edc94b74619a22c-795c3440189a488ea97073a9f2e5856f on tablet 795c3440189a488ea97073a9f2e5856f with peer 87578246c04b49188edc94b74619a22c
I20240502 01:58:12.397728 22910 tablet_bootstrap.cc:492] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.398020 22910 tablet_bootstrap.cc:492] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Bootstrap complete.
I20240502 01:58:12.398295 22910 ts_tablet_manager.cc:1367] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240502 01:58:12.400199 22910 raft_consensus.cc:348] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } }
I20240502 01:58:12.400560 22910 raft_consensus.cc:729] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8690f48d41224e379f2c6c0f25251d4b, State: Initialized, Role: LEARNER
I20240502 01:58:12.400851 22910 consensus_queue.cc:260] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } }
I20240502 01:58:12.401785 22926 ts_tablet_manager.cc:902] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Initiating tablet copy from peer 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:12.402307 22910 ts_tablet_manager.cc:1398] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:12.402493 22926 tablet_copy_client.cc:308] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:36375
I20240502 01:58:12.403563 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session 8690f48d41224e379f2c6c0f25251d4b-56b2393c70cb4dcfbc7e8ba06e67124b received from {username='slave'} at 127.17.243.3:58795
I20240502 01:58:12.403636 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session 8690f48d41224e379f2c6c0f25251d4b-56b2393c70cb4dcfbc7e8ba06e67124b on tablet 56b2393c70cb4dcfbc7e8ba06e67124b with peer 8690f48d41224e379f2c6c0f25251d4b
I20240502 01:58:12.407058 21105 tablet_copy_service.cc:140] P 580cdba303094f18bc3a325acb7095ed: Received BeginTabletCopySession request for tablet 3a5c72f9e6e34cae985bce2cab15cfd4 from peer 998441e608064fad81a173c929277899 ({username='slave'} at 127.17.243.7:38189)
I20240502 01:58:12.407181 21105 tablet_copy_service.cc:161] P 580cdba303094f18bc3a325acb7095ed: Beginning new tablet copy session on tablet 3a5c72f9e6e34cae985bce2cab15cfd4 from peer 998441e608064fad81a173c929277899 at {username='slave'} at 127.17.243.7:38189: session id = 998441e608064fad81a173c929277899-3a5c72f9e6e34cae985bce2cab15cfd4
I20240502 01:58:12.408145 21105 tablet_copy_source_session.cc:205] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.411074 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c72f9e6e34cae985bce2cab15cfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.416368 22926 tablet_copy_client.cc:783] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.416540 22926 tablet_copy_client.cc:647] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.418036 22926 tablet_copy_client.cc:523] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.419420 22926 tablet_bootstrap.cc:492] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Bootstrap starting.
I20240502 01:58:12.423993 22926 log.cc:824] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:12.445137 22926 tablet_bootstrap.cc:492] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.445369 22926 tablet_bootstrap.cc:492] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Bootstrap complete.
I20240502 01:58:12.445600 22926 ts_tablet_manager.cc:1367] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Time spent bootstrapping tablet: real 0.026s user 0.027s sys 0.000s
I20240502 01:58:12.447476 22926 raft_consensus.cc:348] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } }
I20240502 01:58:12.447661 22926 raft_consensus.cc:729] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 998441e608064fad81a173c929277899, State: Initialized, Role: LEARNER
I20240502 01:58:12.447964 22926 consensus_queue.cc:260] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } }
I20240502 01:58:12.448537 22650 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:12.448720 22926 ts_tablet_manager.cc:1398] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:12.449290 21105 tablet_copy_service.cc:342] P 580cdba303094f18bc3a325acb7095ed: Request end of tablet copy session 998441e608064fad81a173c929277899-3a5c72f9e6e34cae985bce2cab15cfd4 received from {username='slave'} at 127.17.243.7:38189
I20240502 01:58:12.449371 21105 tablet_copy_service.cc:433] P 580cdba303094f18bc3a325acb7095ed: ending tablet copy session 998441e608064fad81a173c929277899-3a5c72f9e6e34cae985bce2cab15cfd4 on tablet 3a5c72f9e6e34cae985bce2cab15cfd4 with peer 998441e608064fad81a173c929277899
I20240502 01:58:12.468219 22894 ts_tablet_manager.cc:902] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: Initiating tablet copy from peer 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:12.468591 22894 tablet_copy_client.cc:308] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:36375
I20240502 01:58:12.471892 21105 tablet_copy_service.cc:140] P 580cdba303094f18bc3a325acb7095ed: Received BeginTabletCopySession request for tablet cad36bda76a24e3da0bcd937b30d48a1 from peer c1449418fd3344ce996bf17e7bade3b7 ({username='slave'} at 127.17.243.6:34941)
I20240502 01:58:12.471992 21105 tablet_copy_service.cc:161] P 580cdba303094f18bc3a325acb7095ed: Beginning new tablet copy session on tablet cad36bda76a24e3da0bcd937b30d48a1 from peer c1449418fd3344ce996bf17e7bade3b7 at {username='slave'} at 127.17.243.6:34941: session id = c1449418fd3344ce996bf17e7bade3b7-cad36bda76a24e3da0bcd937b30d48a1
I20240502 01:58:12.472399 22888 ts_tablet_manager.cc:902] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: Initiating tablet copy from peer 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:12.472780 22888 tablet_copy_client.cc:308] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:36375
I20240502 01:58:12.472784 21105 tablet_copy_source_session.cc:205] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.473635 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad36bda76a24e3da0bcd937b30d48a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.475268 22894 tablet_copy_client.cc:783] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.475395 22894 tablet_copy_client.cc:647] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.476285 22894 tablet_copy_client.cc:523] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.477329 22894 tablet_bootstrap.cc:492] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: Bootstrap starting.
I20240502 01:58:12.478124 21105 tablet_copy_service.cc:140] P 580cdba303094f18bc3a325acb7095ed: Received BeginTabletCopySession request for tablet c88b79b757d8428a89debee2187ed755 from peer 63b02225f2b444f29dcf0041105f4784 ({username='slave'} at 127.17.243.5:36361)
I20240502 01:58:12.478202 21105 tablet_copy_service.cc:161] P 580cdba303094f18bc3a325acb7095ed: Beginning new tablet copy session on tablet c88b79b757d8428a89debee2187ed755 from peer 63b02225f2b444f29dcf0041105f4784 at {username='slave'} at 127.17.243.5:36361: session id = 63b02225f2b444f29dcf0041105f4784-c88b79b757d8428a89debee2187ed755
I20240502 01:58:12.478765 21105 tablet_copy_source_session.cc:205] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.479472 22888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88b79b757d8428a89debee2187ed755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.481375 22888 tablet_copy_client.cc:783] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.481511 22888 tablet_copy_client.cc:647] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.482517 22888 tablet_copy_client.cc:523] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.483582 22888 tablet_bootstrap.cc:492] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: Bootstrap starting.
I20240502 01:58:12.489800 22926 ts_tablet_manager.cc:902] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:12.490202 22926 tablet_copy_client.cc:308] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:12.493182 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet 5fbd475017a14bf8b488f2830e2bcee9 from peer 998441e608064fad81a173c929277899 ({username='slave'} at 127.17.243.7:41997)
I20240502 01:58:12.493269 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet 5fbd475017a14bf8b488f2830e2bcee9 from peer 998441e608064fad81a173c929277899 at {username='slave'} at 127.17.243.7:41997: session id = 998441e608064fad81a173c929277899-5fbd475017a14bf8b488f2830e2bcee9
I20240502 01:58:12.493881 20974 tablet_copy_source_session.cc:205] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.494565 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbd475017a14bf8b488f2830e2bcee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.496037 22926 tablet_copy_client.cc:783] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.496158 22926 tablet_copy_client.cc:647] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.497028 22926 tablet_copy_client.cc:523] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.497881 22926 tablet_bootstrap.cc:492] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: Bootstrap starting.
I20240502 01:58:12.501667 22894 tablet_bootstrap.cc:492] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.501888 22894 tablet_bootstrap.cc:492] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: Bootstrap complete.
I20240502 01:58:12.502065 22894 ts_tablet_manager.cc:1367] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.005s
I20240502 01:58:12.502571 22894 raft_consensus.cc:348] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } }
I20240502 01:58:12.502681 22894 raft_consensus.cc:729] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c1449418fd3344ce996bf17e7bade3b7, State: Initialized, Role: LEARNER
I20240502 01:58:12.502774 22894 consensus_queue.cc:260] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: true } }
I20240502 01:58:12.503067 22894 ts_tablet_manager.cc:1398] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:12.503568 21105 tablet_copy_service.cc:342] P 580cdba303094f18bc3a325acb7095ed: Request end of tablet copy session c1449418fd3344ce996bf17e7bade3b7-cad36bda76a24e3da0bcd937b30d48a1 received from {username='slave'} at 127.17.243.6:34941
I20240502 01:58:12.503636 21105 tablet_copy_service.cc:433] P 580cdba303094f18bc3a325acb7095ed: ending tablet copy session c1449418fd3344ce996bf17e7bade3b7-cad36bda76a24e3da0bcd937b30d48a1 on tablet cad36bda76a24e3da0bcd937b30d48a1 with peer c1449418fd3344ce996bf17e7bade3b7
I20240502 01:58:12.507812 22888 tablet_bootstrap.cc:492] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.508004 22888 tablet_bootstrap.cc:492] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: Bootstrap complete.
I20240502 01:58:12.508128 22888 ts_tablet_manager.cc:1367] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240502 01:58:12.508518 22888 raft_consensus.cc:348] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } }
I20240502 01:58:12.508610 22888 raft_consensus.cc:729] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63b02225f2b444f29dcf0041105f4784, State: Initialized, Role: LEARNER
I20240502 01:58:12.508679 22888 consensus_queue.cc:260] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: true } }
I20240502 01:58:12.509459 22904 ts_tablet_manager.cc:902] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: Initiating tablet copy from peer 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:12.509475 22888 ts_tablet_manager.cc:1398] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:58:12.509851 22904 tablet_copy_client.cc:308] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:36375
I20240502 01:58:12.510046 21105 tablet_copy_service.cc:342] P 580cdba303094f18bc3a325acb7095ed: Request end of tablet copy session 63b02225f2b444f29dcf0041105f4784-c88b79b757d8428a89debee2187ed755 received from {username='slave'} at 127.17.243.5:36361
I20240502 01:58:12.510104 21105 tablet_copy_service.cc:433] P 580cdba303094f18bc3a325acb7095ed: ending tablet copy session 63b02225f2b444f29dcf0041105f4784-c88b79b757d8428a89debee2187ed755 on tablet c88b79b757d8428a89debee2187ed755 with peer 63b02225f2b444f29dcf0041105f4784
I20240502 01:58:12.513146 21105 tablet_copy_service.cc:140] P 580cdba303094f18bc3a325acb7095ed: Received BeginTabletCopySession request for tablet 79511d04d45d4c00821f7bcc3231c1a9 from peer 87578246c04b49188edc94b74619a22c ({username='slave'} at 127.17.243.4:60461)
I20240502 01:58:12.513228 21105 tablet_copy_service.cc:161] P 580cdba303094f18bc3a325acb7095ed: Beginning new tablet copy session on tablet 79511d04d45d4c00821f7bcc3231c1a9 from peer 87578246c04b49188edc94b74619a22c at {username='slave'} at 127.17.243.4:60461: session id = 87578246c04b49188edc94b74619a22c-79511d04d45d4c00821f7bcc3231c1a9
I20240502 01:58:12.513823 21105 tablet_copy_source_session.cc:205] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:12.514546 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79511d04d45d4c00821f7bcc3231c1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:12.515704 22926 tablet_bootstrap.cc:492] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.515887 22926 tablet_bootstrap.cc:492] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: Bootstrap complete.
I20240502 01:58:12.516016 22926 ts_tablet_manager.cc:1367] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240502 01:58:12.516173 22904 tablet_copy_client.cc:783] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:12.516300 22904 tablet_copy_client.cc:647] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:12.516443 22926 raft_consensus.cc:348] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } }
I20240502 01:58:12.516541 22926 raft_consensus.cc:729] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 998441e608064fad81a173c929277899, State: Initialized, Role: LEARNER
I20240502 01:58:12.516623 22926 consensus_queue.cc:260] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: true } }
I20240502 01:58:12.516831 22926 ts_tablet_manager.cc:1398] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:12.517320 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session 998441e608064fad81a173c929277899-5fbd475017a14bf8b488f2830e2bcee9 received from {username='slave'} at 127.17.243.7:41997
I20240502 01:58:12.517395 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session 998441e608064fad81a173c929277899-5fbd475017a14bf8b488f2830e2bcee9 on tablet 5fbd475017a14bf8b488f2830e2bcee9 with peer 998441e608064fad81a173c929277899
I20240502 01:58:12.517575 22904 tablet_copy_client.cc:523] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.518543 22904 tablet_bootstrap.cc:492] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: Bootstrap starting.
I20240502 01:58:12.535122 22904 tablet_bootstrap.cc:492] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.535308 22904 tablet_bootstrap.cc:492] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: Bootstrap complete.
I20240502 01:58:12.535432 22904 ts_tablet_manager.cc:1367] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240502 01:58:12.535795 22904 raft_consensus.cc:348] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } }
I20240502 01:58:12.535878 22904 raft_consensus.cc:729] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 87578246c04b49188edc94b74619a22c, State: Initialized, Role: LEARNER
I20240502 01:58:12.535965 22904 consensus_queue.cc:260] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: true } }
I20240502 01:58:12.536159 22904 ts_tablet_manager.cc:1398] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:12.536572 21105 tablet_copy_service.cc:342] P 580cdba303094f18bc3a325acb7095ed: Request end of tablet copy session 87578246c04b49188edc94b74619a22c-79511d04d45d4c00821f7bcc3231c1a9 received from {username='slave'} at 127.17.243.4:60461
I20240502 01:58:12.536640 21105 tablet_copy_service.cc:433] P 580cdba303094f18bc3a325acb7095ed: ending tablet copy session 87578246c04b49188edc94b74619a22c-79511d04d45d4c00821f7bcc3231c1a9 on tablet 79511d04d45d4c00821f7bcc3231c1a9 with peer 87578246c04b49188edc94b74619a22c
I20240502 01:58:12.743038 22341 raft_consensus.cc:1203] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:12.844209 22210 raft_consensus.cc:1203] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:12.851351 22472 raft_consensus.cc:1203] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:12.869832 22604 raft_consensus.cc:1203] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:12.882629 22341 raft_consensus.cc:1203] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:12.926887 22604 raft_consensus.cc:1203] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:12.960929 22079 raft_consensus.cc:1203] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:13.016057 22210 raft_consensus.cc:1203] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:13.039705 22472 raft_consensus.cc:1203] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:13.264457 23040 raft_consensus.cc:1050] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER 87578246c04b49188edc94b74619a22c to VOTER
I20240502 01:58:13.264915 23040 consensus_queue.cc:237] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:13.266016 22210 raft_consensus.cc:1261] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.266438 22753 consensus_queue.cc:1035] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.268811 23040 raft_consensus.cc:2932] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 87578246c04b49188edc94b74619a22c (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } } }
I20240502 01:58:13.269266 22210 raft_consensus.cc:2932] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 87578246c04b49188edc94b74619a22c (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } } }
I20240502 01:58:13.271013 20808 catalog_manager.cc:5484] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, 87578246c04b49188edc94b74619a22c (127.17.243.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.332758 22753 raft_consensus.cc:1050] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER 63b02225f2b444f29dcf0041105f4784 to VOTER
I20240502 01:58:13.333060 22753 consensus_queue.cc:237] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:13.334187 22341 raft_consensus.cc:1261] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.334586 22753 consensus_queue.cc:1035] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.337000 23040 raft_consensus.cc:2932] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } } }
I20240502 01:58:13.337509 22341 raft_consensus.cc:2932] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } } }
I20240502 01:58:13.339062 20810 catalog_manager.cc:5484] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.343266 23039 raft_consensus.cc:1050] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER 998441e608064fad81a173c929277899 to VOTER
I20240502 01:58:13.343515 23039 consensus_queue.cc:237] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:13.344488 22604 raft_consensus.cc:1261] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.344851 23040 consensus_queue.cc:1035] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.347121 23039 raft_consensus.cc:2932] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 998441e608064fad81a173c929277899 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } } }
I20240502 01:58:13.347754 22604 raft_consensus.cc:2932] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 998441e608064fad81a173c929277899 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } } }
I20240502 01:58:13.348997 20808 catalog_manager.cc:5484] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, 998441e608064fad81a173c929277899 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.364799 22753 raft_consensus.cc:1050] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER 8690f48d41224e379f2c6c0f25251d4b to VOTER
I20240502 01:58:13.365049 22753 consensus_queue.cc:237] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:13.366031 22079 raft_consensus.cc:1261] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.366384 22753 consensus_queue.cc:1035] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.368376 23040 raft_consensus.cc:2932] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } } }
I20240502 01:58:13.368834 22079 raft_consensus.cc:2932] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } } }
I20240502 01:58:13.370153 20808 catalog_manager.cc:5484] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.370740 23039 raft_consensus.cc:1050] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER c1449418fd3344ce996bf17e7bade3b7 to VOTER
I20240502 01:58:13.371001 23039 consensus_queue.cc:237] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:13.372234 22472 raft_consensus.cc:1261] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.372604 22753 consensus_queue.cc:1035] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.374614 23039 raft_consensus.cc:2932] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } } }
I20240502 01:58:13.375015 22472 raft_consensus.cc:2932] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } } }
I20240502 01:58:13.376228 20810 catalog_manager.cc:5484] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.388199 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4fd077b7bcc43478ba93f21de746397"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58962
I20240502 01:58:13.388324 20953 raft_consensus.cc:595] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.405550 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8c274b327e04a85b39ef8984a6f486f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58984
I20240502 01:58:13.405640 20953 raft_consensus.cc:595] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.413717 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "795c3440189a488ea97073a9f2e5856f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59004
I20240502 01:58:13.413810 20953 raft_consensus.cc:595] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.427029 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fbd475017a14bf8b488f2830e2bcee9"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59022
I20240502 01:58:13.427124 20953 raft_consensus.cc:595] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.439760 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56b2393c70cb4dcfbc7e8ba06e67124b"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59038
I20240502 01:58:13.439853 20953 raft_consensus.cc:595] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.468854 22924 raft_consensus.cc:1050] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: attempting to promote NON_VOTER 998441e608064fad81a173c929277899 to VOTER
I20240502 01:58:13.469480 22924 consensus_queue.cc:237] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:13.470562 22604 raft_consensus.cc:1261] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 LEARNER]: Refusing update from remote peer 580cdba303094f18bc3a325acb7095ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.470959 23042 consensus_queue.cc:1035] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.472592 22924 raft_consensus.cc:2932] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 998441e608064fad81a173c929277899 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } } }
I20240502 01:58:13.473035 22604 raft_consensus.cc:2932] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 998441e608064fad81a173c929277899 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } } }
I20240502 01:58:13.474766 20808 catalog_manager.cc:5484] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index 4 to 5, 998441e608064fad81a173c929277899 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.478772 23042 raft_consensus.cc:1050] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: attempting to promote NON_VOTER 63b02225f2b444f29dcf0041105f4784 to VOTER
I20240502 01:58:13.479005 23042 consensus_queue.cc:237] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:13.479897 22341 raft_consensus.cc:1261] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 LEARNER]: Refusing update from remote peer 580cdba303094f18bc3a325acb7095ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.480247 23042 consensus_queue.cc:1035] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.480580 23042 raft_consensus.cc:1050] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: attempting to promote NON_VOTER 87578246c04b49188edc94b74619a22c to VOTER
I20240502 01:58:13.480813 23042 consensus_queue.cc:237] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:13.481670 22210 raft_consensus.cc:1261] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 LEARNER]: Refusing update from remote peer 580cdba303094f18bc3a325acb7095ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.481983 23042 raft_consensus.cc:2932] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } } }
I20240502 01:58:13.482295 22341 raft_consensus.cc:2932] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } } }
I20240502 01:58:13.482837 23042 consensus_queue.cc:1035] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.484017 20810 catalog_manager.cc:5484] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index 4 to 5, 63b02225f2b444f29dcf0041105f4784 (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.484980 22924 raft_consensus.cc:2932] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 87578246c04b49188edc94b74619a22c (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } } }
I20240502 01:58:13.485373 22210 raft_consensus.cc:2932] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 87578246c04b49188edc94b74619a22c (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } } }
I20240502 01:58:13.486922 20808 catalog_manager.cc:5484] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index 4 to 5, 87578246c04b49188edc94b74619a22c (127.17.243.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.649647 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4fd077b7bcc43478ba93f21de746397"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59060
I20240502 01:58:13.649771 20953 raft_consensus.cc:595] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.659015 21085 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79511d04d45d4c00821f7bcc3231c1a9"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55398
I20240502 01:58:13.659150 21085 raft_consensus.cc:595] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.659536 23042 raft_consensus.cc:1050] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: attempting to promote NON_VOTER c1449418fd3344ce996bf17e7bade3b7 to VOTER
I20240502 01:58:13.659849 23042 consensus_queue.cc:237] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:13.660859 22472 raft_consensus.cc:1261] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 LEARNER]: Refusing update from remote peer 580cdba303094f18bc3a325acb7095ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.661248 23042 consensus_queue.cc:1035] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.662947 23041 raft_consensus.cc:2932] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } } }
I20240502 01:58:13.663350 22472 raft_consensus.cc:2932] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } } }
I20240502 01:58:13.665727 20810 catalog_manager.cc:5484] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index 4 to 5, c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.669771 21085 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c88b79b757d8428a89debee2187ed755"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55402
I20240502 01:58:13.669865 21085 raft_consensus.cc:595] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.678665 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8c274b327e04a85b39ef8984a6f486f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59074
I20240502 01:58:13.678768 20953 raft_consensus.cc:595] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.687424 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "795c3440189a488ea97073a9f2e5856f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59090
I20240502 01:58:13.687518 20953 raft_consensus.cc:595] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.695986 21085 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a5c72f9e6e34cae985bce2cab15cfd4"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55420
I20240502 01:58:13.696079 21085 raft_consensus.cc:595] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.704699 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fbd475017a14bf8b488f2830e2bcee9"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59104
I20240502 01:58:13.704804 20953 raft_consensus.cc:595] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.713754 21085 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cad36bda76a24e3da0bcd937b30d48a1"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55432
I20240502 01:58:13.713845 21085 raft_consensus.cc:595] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.722983 20953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56b2393c70cb4dcfbc7e8ba06e67124b"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59120
I20240502 01:58:13.723090 20953 raft_consensus.cc:595] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.723975 22753 raft_consensus.cc:979] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: : Instructing follower 87578246c04b49188edc94b74619a22c to start an election
I20240502 01:58:13.724093 22753 raft_consensus.cc:1067] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer 87578246c04b49188edc94b74619a22c to start an election
I20240502 01:58:13.724668 22210 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "795c3440189a488ea97073a9f2e5856f"
dest_uuid: "87578246c04b49188edc94b74619a22c"
from {username='slave'} at 127.17.243.1:59255
I20240502 01:58:13.724866 22210 raft_consensus.cc:482] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.724983 22210 raft_consensus.cc:3037] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.755383 22210 raft_consensus.cc:504] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:13.756376 22210 leader_election.cc:288] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:13.756840 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795c3440189a488ea97073a9f2e5856f" candidate_uuid: "87578246c04b49188edc94b74619a22c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:13.757035 20953 raft_consensus.cc:3032] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:13.757088 20953 raft_consensus.cc:729] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:13.757256 20953 consensus_queue.cc:260] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:13.757424 20953 raft_consensus.cc:3037] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.758477 20953 raft_consensus.cc:2454] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87578246c04b49188edc94b74619a22c in term 2.
I20240502 01:58:13.758899 22144 leader_election.cc:302] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, 87578246c04b49188edc94b74619a22c; no voters:
I20240502 01:58:13.759140 22918 raft_consensus.cc:2781] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:13.759315 22918 raft_consensus.cc:686] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 2 LEADER]: Becoming Leader. State: Replica: 87578246c04b49188edc94b74619a22c, State: Running, Role: LEADER
I20240502 01:58:13.759593 22918 consensus_queue.cc:237] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:13.761430 20808 catalog_manager.cc:5484] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to 87578246c04b49188edc94b74619a22c (127.17.243.4). New cstate: current_term: 2 leader_uuid: "87578246c04b49188edc94b74619a22c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:13.767174 20797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 795c3440189a488ea97073a9f2e5856f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 87578246c04b49188edc94b74619a22c (127.17.243.4:44049) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:58:13.846794 23040 raft_consensus.cc:979] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: : Instructing follower 63b02225f2b444f29dcf0041105f4784 to start an election
I20240502 01:58:13.846918 23040 raft_consensus.cc:1067] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer 63b02225f2b444f29dcf0041105f4784 to start an election
I20240502 01:58:13.847488 22341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8c274b327e04a85b39ef8984a6f486f"
dest_uuid: "63b02225f2b444f29dcf0041105f4784"
from {username='slave'} at 127.17.243.1:47939
I20240502 01:58:13.847687 22341 raft_consensus.cc:482] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.847792 22341 raft_consensus.cc:3037] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.850477 22753 raft_consensus.cc:979] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: : Instructing follower 8690f48d41224e379f2c6c0f25251d4b to start an election
I20240502 01:58:13.850569 22753 raft_consensus.cc:1067] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer 8690f48d41224e379f2c6c0f25251d4b to start an election
I20240502 01:58:13.851009 22079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "56b2393c70cb4dcfbc7e8ba06e67124b"
dest_uuid: "8690f48d41224e379f2c6c0f25251d4b"
from {username='slave'} at 127.17.243.1:37839
I20240502 01:58:13.851171 22079 raft_consensus.cc:482] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.851254 22079 raft_consensus.cc:3037] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.858827 23040 raft_consensus.cc:979] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: : Instructing follower c1449418fd3344ce996bf17e7bade3b7 to start an election
I20240502 01:58:13.858922 22753 raft_consensus.cc:1067] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer c1449418fd3344ce996bf17e7bade3b7 to start an election
I20240502 01:58:13.859331 22472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4fd077b7bcc43478ba93f21de746397"
dest_uuid: "c1449418fd3344ce996bf17e7bade3b7"
from {username='slave'} at 127.17.243.1:46465
I20240502 01:58:13.859488 22472 raft_consensus.cc:482] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.859562 22472 raft_consensus.cc:3037] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.930147 22472 raft_consensus.cc:504] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:13.930227 22079 raft_consensus.cc:504] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:13.931061 22472 leader_election.cc:288] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:13.931097 22079 leader_election.cc:288] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:13.931478 20953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fd077b7bcc43478ba93f21de746397" candidate_uuid: "c1449418fd3344ce996bf17e7bade3b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:13.931596 20953 raft_consensus.cc:3032] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:13.931635 20953 raft_consensus.cc:729] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:13.931771 20953 consensus_queue.cc:260] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:13.931875 20953 raft_consensus.cc:3037] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.931915 22341 raft_consensus.cc:504] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:13.932112 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2393c70cb4dcfbc7e8ba06e67124b" candidate_uuid: "8690f48d41224e379f2c6c0f25251d4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:13.932209 20952 raft_consensus.cc:3032] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:13.932246 20952 raft_consensus.cc:729] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:13.932356 20952 consensus_queue.cc:260] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:13.932473 20952 raft_consensus.cc:3037] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.932641 22341 leader_election.cc:288] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:13.933212 20951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c274b327e04a85b39ef8984a6f486f" candidate_uuid: "63b02225f2b444f29dcf0041105f4784" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:13.933310 20951 raft_consensus.cc:3032] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:13.933351 20951 raft_consensus.cc:729] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:13.933472 20951 consensus_queue.cc:260] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:13.933579 20951 raft_consensus.cc:3037] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.934887 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4fd077b7bcc43478ba93f21de746397"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59142
I20240502 01:58:13.937537 23041 raft_consensus.cc:979] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: : Instructing follower 63b02225f2b444f29dcf0041105f4784 to start an election
I20240502 01:58:13.937628 23041 raft_consensus.cc:1067] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Signalling peer 63b02225f2b444f29dcf0041105f4784 to start an election
I20240502 01:58:13.938109 22341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c88b79b757d8428a89debee2187ed755"
dest_uuid: "63b02225f2b444f29dcf0041105f4784"
from {username='slave'} at 127.17.243.2:44353
I20240502 01:58:13.938211 22341 raft_consensus.cc:482] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.938256 22341 raft_consensus.cc:3037] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.956452 23040 raft_consensus.cc:979] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: : Instructing follower 998441e608064fad81a173c929277899 to start an election
I20240502 01:58:13.956547 23040 raft_consensus.cc:1067] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer 998441e608064fad81a173c929277899 to start an election
I20240502 01:58:13.956949 22604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbd475017a14bf8b488f2830e2bcee9"
dest_uuid: "998441e608064fad81a173c929277899"
from {username='slave'} at 127.17.243.1:45603
I20240502 01:58:13.957108 22604 raft_consensus.cc:482] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.957199 22604 raft_consensus.cc:3037] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.960480 22924 raft_consensus.cc:979] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: : Instructing follower 87578246c04b49188edc94b74619a22c to start an election
I20240502 01:58:13.960609 22924 raft_consensus.cc:1067] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Signalling peer 87578246c04b49188edc94b74619a22c to start an election
I20240502 01:58:13.960995 22210 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79511d04d45d4c00821f7bcc3231c1a9"
dest_uuid: "87578246c04b49188edc94b74619a22c"
from {username='slave'} at 127.17.243.2:56185
I20240502 01:58:13.961104 22210 raft_consensus.cc:482] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.961151 22210 raft_consensus.cc:3037] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.026144 20953 raft_consensus.cc:2454] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1449418fd3344ce996bf17e7bade3b7 in term 2.
I20240502 01:58:14.026154 22341 raft_consensus.cc:504] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:14.026165 22604 raft_consensus.cc:504] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.026314 20950 raft_consensus.cc:595] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 01:58:14.026358 20950 raft_consensus.cc:603] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 01:58:14.026540 22341 leader_election.cc:288] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [CANDIDATE]: Term 2 election: Requested vote from peers 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:14.026808 22210 raft_consensus.cc:504] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:14.027083 22604 leader_election.cc:288] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:14.027163 22210 leader_election.cc:288] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [CANDIDATE]: Term 2 election: Requested vote from peers 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:14.027276 21085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88b79b757d8428a89debee2187ed755" candidate_uuid: "63b02225f2b444f29dcf0041105f4784" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "580cdba303094f18bc3a325acb7095ed"
I20240502 01:58:14.027426 21085 raft_consensus.cc:3032] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:14.027467 21085 raft_consensus.cc:729] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:14.027621 21085 consensus_queue.cc:260] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:14.027669 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79511d04d45d4c00821f7bcc3231c1a9" candidate_uuid: "87578246c04b49188edc94b74619a22c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "580cdba303094f18bc3a325acb7095ed"
I20240502 01:58:14.027741 21084 raft_consensus.cc:3032] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:14.027781 21084 raft_consensus.cc:729] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:14.027909 21084 consensus_queue.cc:260] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:14.027784 21085 raft_consensus.cc:3037] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.028031 21084 raft_consensus.cc:3037] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.028306 20952 raft_consensus.cc:2454] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8690f48d41224e379f2c6c0f25251d4b in term 2.
I20240502 01:58:14.028544 20951 raft_consensus.cc:2454] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b02225f2b444f29dcf0041105f4784 in term 2.
I20240502 01:58:14.028805 22406 leader_election.cc:302] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, c1449418fd3344ce996bf17e7bade3b7; no voters:
I20240502 01:58:14.028926 21085 raft_consensus.cc:2454] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b02225f2b444f29dcf0041105f4784 in term 2.
I20240502 01:58:14.028987 21084 raft_consensus.cc:2454] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87578246c04b49188edc94b74619a22c in term 2.
I20240502 01:58:14.029170 22903 raft_consensus.cc:2781] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.029343 22903 raft_consensus.cc:686] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 2 LEADER]: Becoming Leader. State: Replica: c1449418fd3344ce996bf17e7bade3b7, State: Running, Role: LEADER
I20240502 01:58:14.029450 22013 leader_election.cc:302] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, 8690f48d41224e379f2c6c0f25251d4b; no voters:
I20240502 01:58:14.029453 22275 leader_election.cc:302] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, 63b02225f2b444f29dcf0041105f4784; no voters:
I20240502 01:58:14.029626 22903 consensus_queue.cc:237] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:14.029824 22275 leader_election.cc:302] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed, 63b02225f2b444f29dcf0041105f4784; no voters:
I20240502 01:58:14.029883 23331 raft_consensus.cc:2781] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.030059 23331 raft_consensus.cc:686] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 2 LEADER]: Becoming Leader. State: Replica: 63b02225f2b444f29dcf0041105f4784, State: Running, Role: LEADER
I20240502 01:58:14.030139 22144 leader_election.cc:302] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed, 87578246c04b49188edc94b74619a22c; no voters:
I20240502 01:58:14.030192 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbd475017a14bf8b488f2830e2bcee9" candidate_uuid: "998441e608064fad81a173c929277899" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:14.030287 20950 raft_consensus.cc:3032] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:14.030321 20950 raft_consensus.cc:729] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:14.030328 22918 raft_consensus.cc:2781] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.030300 23331 consensus_queue.cc:237] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:14.030388 22918 raft_consensus.cc:686] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 2 LEADER]: Becoming Leader. State: Replica: 87578246c04b49188edc94b74619a22c, State: Running, Role: LEADER
I20240502 01:58:14.030463 20950 consensus_queue.cc:260] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.030508 22918 consensus_queue.cc:237] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:14.030592 20950 raft_consensus.cc:3037] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.031463 23333 raft_consensus.cc:2781] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.031692 23333 raft_consensus.cc:686] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 2 LEADER]: Becoming Leader. State: Replica: 63b02225f2b444f29dcf0041105f4784, State: Running, Role: LEADER
I20240502 01:58:14.031877 23333 consensus_queue.cc:237] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:14.032553 20808 catalog_manager.cc:5484] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c reported cstate change: term changed from 1 to 2, leader changed from 580cdba303094f18bc3a325acb7095ed (127.17.243.2) to 87578246c04b49188edc94b74619a22c (127.17.243.4). New cstate: current_term: 2 leader_uuid: "87578246c04b49188edc94b74619a22c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.032723 23330 raft_consensus.cc:2781] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.032894 23330 raft_consensus.cc:686] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 2 LEADER]: Becoming Leader. State: Replica: 8690f48d41224e379f2c6c0f25251d4b, State: Running, Role: LEADER
I20240502 01:58:14.033265 20810 catalog_manager.cc:5484] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6). New cstate: current_term: 2 leader_uuid: "c1449418fd3344ce996bf17e7bade3b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.033357 23330 consensus_queue.cc:237] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:14.035830 20808 catalog_manager.cc:5484] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3). New cstate: current_term: 2 leader_uuid: "8690f48d41224e379f2c6c0f25251d4b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.036270 20810 catalog_manager.cc:5484] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to 63b02225f2b444f29dcf0041105f4784 (127.17.243.5). New cstate: current_term: 2 leader_uuid: "63b02225f2b444f29dcf0041105f4784" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.040635 20810 catalog_manager.cc:5484] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 reported cstate change: term changed from 1 to 2, leader changed from 580cdba303094f18bc3a325acb7095ed (127.17.243.2) to 63b02225f2b444f29dcf0041105f4784 (127.17.243.5). New cstate: current_term: 2 leader_uuid: "63b02225f2b444f29dcf0041105f4784" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.064689 21084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a5c72f9e6e34cae985bce2cab15cfd4"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55450
I20240502 01:58:14.064802 21084 raft_consensus.cc:595] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:14.084717 23041 raft_consensus.cc:979] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: : Instructing follower 998441e608064fad81a173c929277899 to start an election
I20240502 01:58:14.084798 23041 raft_consensus.cc:1067] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Signalling peer 998441e608064fad81a173c929277899 to start an election
I20240502 01:58:14.085215 22604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a5c72f9e6e34cae985bce2cab15cfd4"
dest_uuid: "998441e608064fad81a173c929277899"
from {username='slave'} at 127.17.243.2:50357
I20240502 01:58:14.085330 22604 raft_consensus.cc:482] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:14.085393 22604 raft_consensus.cc:3037] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.120539 20950 raft_consensus.cc:2454] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998441e608064fad81a173c929277899 in term 2.
I20240502 01:58:14.120537 22604 raft_consensus.cc:504] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.120887 22604 leader_election.cc:288] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [CANDIDATE]: Term 2 election: Requested vote from peers 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:14.120950 22537 leader_election.cc:302] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, 998441e608064fad81a173c929277899; no voters:
I20240502 01:58:14.121268 23376 raft_consensus.cc:2781] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.121467 23376 raft_consensus.cc:686] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 2 LEADER]: Becoming Leader. State: Replica: 998441e608064fad81a173c929277899, State: Running, Role: LEADER
I20240502 01:58:14.121605 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c72f9e6e34cae985bce2cab15cfd4" candidate_uuid: "998441e608064fad81a173c929277899" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "580cdba303094f18bc3a325acb7095ed"
I20240502 01:58:14.121699 21084 raft_consensus.cc:3032] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:14.121730 21084 raft_consensus.cc:729] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:14.121855 21084 consensus_queue.cc:260] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.122000 21084 raft_consensus.cc:3037] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.122330 23376 consensus_queue.cc:237] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.122849 21084 raft_consensus.cc:2454] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998441e608064fad81a173c929277899 in term 2.
I20240502 01:58:14.123113 22537 leader_election.cc:302] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed, 998441e608064fad81a173c929277899; no voters:
I20240502 01:58:14.123327 23377 raft_consensus.cc:2781] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.123479 23377 raft_consensus.cc:686] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 2 LEADER]: Becoming Leader. State: Replica: 998441e608064fad81a173c929277899, State: Running, Role: LEADER
I20240502 01:58:14.123603 23377 consensus_queue.cc:237] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.124785 20810 catalog_manager.cc:5484] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to 998441e608064fad81a173c929277899 (127.17.243.7). New cstate: current_term: 2 leader_uuid: "998441e608064fad81a173c929277899" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.128461 20810 catalog_manager.cc:5484] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 reported cstate change: term changed from 1 to 2, leader changed from 580cdba303094f18bc3a325acb7095ed (127.17.243.2) to 998441e608064fad81a173c929277899 (127.17.243.7). New cstate: current_term: 2 leader_uuid: "998441e608064fad81a173c929277899" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.132939 21084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cad36bda76a24e3da0bcd937b30d48a1"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55462
I20240502 01:58:14.133038 21084 raft_consensus.cc:595] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:14.176916 22924 raft_consensus.cc:979] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: : Instructing follower c1449418fd3344ce996bf17e7bade3b7 to start an election
I20240502 01:58:14.176996 22924 raft_consensus.cc:1067] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Signalling peer c1449418fd3344ce996bf17e7bade3b7 to start an election
I20240502 01:58:14.177361 22472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cad36bda76a24e3da0bcd937b30d48a1"
dest_uuid: "c1449418fd3344ce996bf17e7bade3b7"
from {username='slave'} at 127.17.243.2:44349
I20240502 01:58:14.177479 22472 raft_consensus.cc:482] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:14.177538 22472 raft_consensus.cc:3037] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.218959 22472 raft_consensus.cc:504] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:14.219347 22472 leader_election.cc:288] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [CANDIDATE]: Term 2 election: Requested vote from peers 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:14.219734 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad36bda76a24e3da0bcd937b30d48a1" candidate_uuid: "c1449418fd3344ce996bf17e7bade3b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "580cdba303094f18bc3a325acb7095ed"
I20240502 01:58:14.219852 21084 raft_consensus.cc:3032] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:14.219885 21084 raft_consensus.cc:729] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:14.220029 21084 consensus_queue.cc:260] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:14.220147 21084 raft_consensus.cc:3037] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:14.221050 21084 raft_consensus.cc:2454] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1449418fd3344ce996bf17e7bade3b7 in term 2.
I20240502 01:58:14.221340 22406 leader_election.cc:302] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed, c1449418fd3344ce996bf17e7bade3b7; no voters:
I20240502 01:58:14.221539 22903 raft_consensus.cc:2781] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:14.221637 22903 raft_consensus.cc:686] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 2 LEADER]: Becoming Leader. State: Replica: c1449418fd3344ce996bf17e7bade3b7, State: Running, Role: LEADER
I20240502 01:58:14.221808 22903 consensus_queue.cc:237] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:14.223250 20810 catalog_manager.cc:5484] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 reported cstate change: term changed from 1 to 2, leader changed from 580cdba303094f18bc3a325acb7095ed (127.17.243.2) to c1449418fd3344ce996bf17e7bade3b7 (127.17.243.6). New cstate: current_term: 2 leader_uuid: "c1449418fd3344ce996bf17e7bade3b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.295315 20951 raft_consensus.cc:1261] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer 87578246c04b49188edc94b74619a22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.295931 22918 consensus_queue.cc:1035] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.412693 22210 consensus_queue.cc:237] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:14.413267 23316 raft_consensus.cc:2932] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } } }
I20240502 01:58:14.415995 20797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 795c3440189a488ea97073a9f2e5856f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.416404 21084 raft_consensus.cc:1261] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Refusing update from remote peer 87578246c04b49188edc94b74619a22c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.416842 20808 catalog_manager.cc:5484] T 795c3440189a488ea97073a9f2e5856f P 87578246c04b49188edc94b74619a22c reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "87578246c04b49188edc94b74619a22c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.416940 22918 consensus_queue.cc:1035] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.419762 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet 795c3440189a488ea97073a9f2e5856f with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:14.420214 23444 tablet_replica.cc:330] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:14.420495 23444 raft_consensus.cc:2229] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.420648 23444 raft_consensus.cc:2258] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.421825 23444 ts_tablet_manager.cc:1875] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.426055 20951 raft_consensus.cc:1261] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer c1449418fd3344ce996bf17e7bade3b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.426499 22903 consensus_queue.cc:1035] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.476794 23444 ts_tablet_manager.cc:1888] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.476886 23444 log.cc:1196] T 795c3440189a488ea97073a9f2e5856f P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/795c3440189a488ea97073a9f2e5856f
I20240502 01:58:14.477399 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet 795c3440189a488ea97073a9f2e5856f (table rebalance_test_table_4 [id=1b1ddb13c1824db89bc83e60ee62397f]) successfully deleted
I20240502 01:58:14.579042 20951 raft_consensus.cc:1261] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer 998441e608064fad81a173c929277899: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.579514 23382 consensus_queue.cc:1035] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.584231 21084 raft_consensus.cc:1261] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Refusing update from remote peer 63b02225f2b444f29dcf0041105f4784: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.584688 23449 consensus_queue.cc:1035] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.640581 20951 consensus_queue.cc:237] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } }
I20240502 01:58:14.641396 23040 raft_consensus.cc:2932] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } } }
I20240502 01:58:14.641594 22210 consensus_queue.cc:237] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } }
I20240502 01:58:14.642194 23316 raft_consensus.cc:2932] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } } }
W20240502 01:58:14.642520 20889 consensus_peers.cc:485] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f -> Peer 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3:35731): Couldn't send request to peer 8690f48d41224e379f2c6c0f25251d4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d024eee194347779dd13f423cc9023c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:14.646451 20951 raft_consensus.cc:1261] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer 8690f48d41224e379f2c6c0f25251d4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.646934 23505 consensus_queue.cc:1035] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.647419 20951 raft_consensus.cc:1261] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer 63b02225f2b444f29dcf0041105f4784: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.647758 23451 consensus_queue.cc:1035] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.661624 20797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79511d04d45d4c00821f7bcc3231c1a9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.661969 20808 catalog_manager.cc:5484] T 79511d04d45d4c00821f7bcc3231c1a9 P 87578246c04b49188edc94b74619a22c reported cstate change: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New cstate: current_term: 2 leader_uuid: "87578246c04b49188edc94b74619a22c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87578246c04b49188edc94b74619a22c" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.662312 21084 raft_consensus.cc:1261] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Refusing update from remote peer 998441e608064fad81a173c929277899: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.662526 20809 catalog_manager.cc:5484] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:14.662703 23376 consensus_queue.cc:1035] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.666213 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet 79511d04d45d4c00821f7bcc3231c1a9 with delete_type TABLET_DATA_TOMBSTONED (TS 580cdba303094f18bc3a325acb7095ed not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50226
I20240502 01:58:14.666883 23519 tablet_replica.cc:330] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: stopping tablet replica
I20240502 01:58:14.667183 23519 raft_consensus.cc:2229] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.667316 23519 raft_consensus.cc:2258] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.668411 23519 ts_tablet_manager.cc:1875] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.668773 22472 consensus_queue.cc:237] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:14.669044 21084 raft_consensus.cc:1261] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Refusing update from remote peer c1449418fd3344ce996bf17e7bade3b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.669418 22903 consensus_queue.cc:1035] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.669418 23329 raft_consensus.cc:2932] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } } }
I20240502 01:58:14.684193 22341 consensus_queue.cc:237] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:14.684398 22079 consensus_queue.cc:237] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:14.684819 23451 raft_consensus.cc:2932] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } } }
I20240502 01:58:14.684952 23509 raft_consensus.cc:2932] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } } }
I20240502 01:58:14.742825 22604 consensus_queue.cc:237] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.743952 23382 raft_consensus.cc:2932] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } } }
I20240502 01:58:14.760020 20795 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4fd077b7bcc43478ba93f21de746397 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.760228 23519 ts_tablet_manager.cc:1888] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.760318 23519 log.cc:1196] T 79511d04d45d4c00821f7bcc3231c1a9 P 580cdba303094f18bc3a325acb7095ed: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/79511d04d45d4c00821f7bcc3231c1a9
I20240502 01:58:14.760883 20796 catalog_manager.cc:4830] TS 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375): tablet 79511d04d45d4c00821f7bcc3231c1a9 (table rebalance_test_table_3 [id=43fc2eb6b2854abb8a75cfbccd8844a2]) successfully deleted
I20240502 01:58:14.760855 20808 catalog_manager.cc:5484] T c4fd077b7bcc43478ba93f21de746397 P c1449418fd3344ce996bf17e7bade3b7 reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "c1449418fd3344ce996bf17e7bade3b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.762037 20798 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56b2393c70cb4dcfbc7e8ba06e67124b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.762298 20797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a5c72f9e6e34cae985bce2cab15cfd4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.763154 20810 catalog_manager.cc:5484] T 56b2393c70cb4dcfbc7e8ba06e67124b P 8690f48d41224e379f2c6c0f25251d4b reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "8690f48d41224e379f2c6c0f25251d4b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.763464 20797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c88b79b757d8428a89debee2187ed755 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.763820 20809 catalog_manager.cc:5484] T c88b79b757d8428a89debee2187ed755 P 63b02225f2b444f29dcf0041105f4784 reported cstate change: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New cstate: current_term: 2 leader_uuid: "63b02225f2b444f29dcf0041105f4784" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.763972 20811 catalog_manager.cc:5484] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 998441e608064fad81a173c929277899 reported cstate change: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New cstate: current_term: 2 leader_uuid: "998441e608064fad81a173c929277899" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.764710 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fd077b7bcc43478ba93f21de746397 with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:14.764914 23444 tablet_replica.cc:330] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:14.765049 23444 raft_consensus.cc:2229] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.765161 23444 raft_consensus.cc:2258] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.765834 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b2393c70cb4dcfbc7e8ba06e67124b with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:14.765866 23444 ts_tablet_manager.cc:1875] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.767326 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet c88b79b757d8428a89debee2187ed755 with delete_type TABLET_DATA_TOMBSTONED (TS 580cdba303094f18bc3a325acb7095ed not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50226
I20240502 01:58:14.767817 23519 tablet_replica.cc:330] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: stopping tablet replica
I20240502 01:58:14.767930 23519 raft_consensus.cc:2229] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.768009 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5c72f9e6e34cae985bce2cab15cfd4 with delete_type TABLET_DATA_TOMBSTONED (TS 580cdba303094f18bc3a325acb7095ed not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50226
I20240502 01:58:14.768033 23519 raft_consensus.cc:2258] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.768651 23519 ts_tablet_manager.cc:1875] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.805516 22604 consensus_queue.cc:237] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } }
I20240502 01:58:14.806196 23377 raft_consensus.cc:2932] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } } }
I20240502 01:58:14.807839 20797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5fbd475017a14bf8b488f2830e2bcee9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.808692 20811 catalog_manager.cc:5484] T 5fbd475017a14bf8b488f2830e2bcee9 P 998441e608064fad81a173c929277899 reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "998441e608064fad81a173c929277899" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "998441e608064fad81a173c929277899" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 33251 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.810905 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fbd475017a14bf8b488f2830e2bcee9 with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:14.820194 23444 ts_tablet_manager.cc:1888] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.820215 23519 ts_tablet_manager.cc:1888] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.820281 23444 log.cc:1196] T c4fd077b7bcc43478ba93f21de746397 P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/c4fd077b7bcc43478ba93f21de746397
I20240502 01:58:14.820281 23519 log.cc:1196] T c88b79b757d8428a89debee2187ed755 P 580cdba303094f18bc3a325acb7095ed: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/c88b79b757d8428a89debee2187ed755
I20240502 01:58:14.820704 23444 tablet_replica.cc:330] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:14.820703 23519 tablet_replica.cc:330] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: stopping tablet replica
I20240502 01:58:14.820808 23444 raft_consensus.cc:2229] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.820808 23519 raft_consensus.cc:2229] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.820896 23519 raft_consensus.cc:2258] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.820896 23444 raft_consensus.cc:2258] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.820868 20796 catalog_manager.cc:4830] TS 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375): tablet c88b79b757d8428a89debee2187ed755 (table rebalance_test_table_1 [id=5508a27a4396491786f2f6fd3b5a8632]) successfully deleted
I20240502 01:58:14.821005 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet c4fd077b7bcc43478ba93f21de746397 (table rebalance_test_table_2 [id=e5eb5c4bfbb140ec9d8781f1ca346fab]) successfully deleted
I20240502 01:58:14.821586 23444 ts_tablet_manager.cc:1875] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.823576 23444 ts_tablet_manager.cc:1888] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.823653 23444 log.cc:1196] T 56b2393c70cb4dcfbc7e8ba06e67124b P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/56b2393c70cb4dcfbc7e8ba06e67124b
I20240502 01:58:14.824050 23444 tablet_replica.cc:330] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:14.824111 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet 56b2393c70cb4dcfbc7e8ba06e67124b (table rebalance_test_table_4 [id=1b1ddb13c1824db89bc83e60ee62397f]) successfully deleted
I20240502 01:58:14.824149 23444 raft_consensus.cc:2229] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.824260 23444 raft_consensus.cc:2258] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.824851 23444 ts_tablet_manager.cc:1875] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.826463 23519 ts_tablet_manager.cc:1875] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.826633 23444 ts_tablet_manager.cc:1888] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.826694 23444 log.cc:1196] T 5fbd475017a14bf8b488f2830e2bcee9 P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/5fbd475017a14bf8b488f2830e2bcee9
I20240502 01:58:14.827108 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet 5fbd475017a14bf8b488f2830e2bcee9 (table rebalance_test_table_0 [id=b99bc3270af143db8e5641075113894b]) successfully deleted
I20240502 01:58:14.828296 23519 ts_tablet_manager.cc:1888] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.828356 23519 log.cc:1196] T 3a5c72f9e6e34cae985bce2cab15cfd4 P 580cdba303094f18bc3a325acb7095ed: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a5c72f9e6e34cae985bce2cab15cfd4
I20240502 01:58:14.828717 20796 catalog_manager.cc:4830] TS 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375): tablet 3a5c72f9e6e34cae985bce2cab15cfd4 (table rebalance_test_table_1 [id=5508a27a4396491786f2f6fd3b5a8632]) successfully deleted
I20240502 01:58:14.845873 22472 consensus_queue.cc:237] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } }
I20240502 01:58:14.846412 23521 raft_consensus.cc:2932] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } } }
I20240502 01:58:14.847414 20795 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cad36bda76a24e3da0bcd937b30d48a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:14.848116 20811 catalog_manager.cc:5484] T cad36bda76a24e3da0bcd937b30d48a1 P c1449418fd3344ce996bf17e7bade3b7 reported cstate change: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New cstate: current_term: 2 leader_uuid: "c1449418fd3344ce996bf17e7bade3b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1449418fd3344ce996bf17e7bade3b7" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.849901 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet cad36bda76a24e3da0bcd937b30d48a1 with delete_type TABLET_DATA_TOMBSTONED (TS 580cdba303094f18bc3a325acb7095ed not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50226
I20240502 01:58:14.850104 23519 tablet_replica.cc:330] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: stopping tablet replica
I20240502 01:58:14.850215 23519 raft_consensus.cc:2229] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.850307 23519 raft_consensus.cc:2258] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.850790 23519 ts_tablet_manager.cc:1875] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.852068 23519 ts_tablet_manager.cc:1888] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.852125 23519 log.cc:1196] T cad36bda76a24e3da0bcd937b30d48a1 P 580cdba303094f18bc3a325acb7095ed: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/cad36bda76a24e3da0bcd937b30d48a1
I20240502 01:58:14.852511 20796 catalog_manager.cc:4830] TS 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375): tablet cad36bda76a24e3da0bcd937b30d48a1 (table rebalance_test_table_3 [id=43fc2eb6b2854abb8a75cfbccd8844a2]) successfully deleted
I20240502 01:58:15.208099 23625 ts_tablet_manager.cc:902] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:15.208576 22341 consensus_queue.cc:237] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } }
I20240502 01:58:15.208652 23625 tablet_copy_client.cc:308] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:15.209105 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet 3d024eee194347779dd13f423cc9023c from peer 8690f48d41224e379f2c6c0f25251d4b ({username='slave'} at 127.17.243.3:58795)
I20240502 01:58:15.209198 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet 3d024eee194347779dd13f423cc9023c from peer 8690f48d41224e379f2c6c0f25251d4b at {username='slave'} at 127.17.243.3:58795: session id = 8690f48d41224e379f2c6c0f25251d4b-3d024eee194347779dd13f423cc9023c
I20240502 01:58:15.209180 23449 raft_consensus.cc:2932] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } } }
I20240502 01:58:15.209885 20974 tablet_copy_source_session.cc:205] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:15.210474 20797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8c274b327e04a85b39ef8984a6f486f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240502 01:58:15.210816 23625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d024eee194347779dd13f423cc9023c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:15.211329 20811 catalog_manager.cc:5484] T a8c274b327e04a85b39ef8984a6f486f P 63b02225f2b444f29dcf0041105f4784 reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "63b02225f2b444f29dcf0041105f4784" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63b02225f2b444f29dcf0041105f4784" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 38921 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:15.212754 23625 tablet_copy_client.cc:783] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:15.212890 23625 tablet_copy_client.cc:647] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:15.213501 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c274b327e04a85b39ef8984a6f486f with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:15.213773 23444 tablet_replica.cc:330] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:15.213899 23444 raft_consensus.cc:2229] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:15.214038 23444 raft_consensus.cc:2258] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:15.214063 23625 tablet_copy_client.cc:523] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:15.214704 23444 ts_tablet_manager.cc:1875] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:15.215183 23625 tablet_bootstrap.cc:492] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: Bootstrap starting.
I20240502 01:58:15.216826 23444 ts_tablet_manager.cc:1888] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:15.216903 23444 log.cc:1196] T a8c274b327e04a85b39ef8984a6f486f P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/a8c274b327e04a85b39ef8984a6f486f
I20240502 01:58:15.217366 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet a8c274b327e04a85b39ef8984a6f486f (table rebalance_test_table_0 [id=b99bc3270af143db8e5641075113894b]) successfully deleted
I20240502 01:58:15.231684 23625 tablet_bootstrap.cc:492] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:15.231886 23625 tablet_bootstrap.cc:492] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: Bootstrap complete.
I20240502 01:58:15.232024 23625 ts_tablet_manager.cc:1367] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240502 01:58:15.232434 23625 raft_consensus.cc:348] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } }
I20240502 01:58:15.232517 23625 raft_consensus.cc:729] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8690f48d41224e379f2c6c0f25251d4b, State: Initialized, Role: LEARNER
I20240502 01:58:15.232602 23625 consensus_queue.cc:260] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: true } }
I20240502 01:58:15.232827 23625 ts_tablet_manager.cc:1398] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:15.233254 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session 8690f48d41224e379f2c6c0f25251d4b-3d024eee194347779dd13f423cc9023c received from {username='slave'} at 127.17.243.3:58795
I20240502 01:58:15.233317 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session 8690f48d41224e379f2c6c0f25251d4b-3d024eee194347779dd13f423cc9023c on tablet 3d024eee194347779dd13f423cc9023c with peer 8690f48d41224e379f2c6c0f25251d4b
I20240502 01:58:15.697654 22079 raft_consensus.cc:1203] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:16.189968 23624 raft_consensus.cc:1050] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER 8690f48d41224e379f2c6c0f25251d4b to VOTER
I20240502 01:58:16.190310 23624 consensus_queue.cc:237] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:16.191359 22079 raft_consensus.cc:1261] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:16.191709 23624 consensus_queue.cc:1035] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:16.193293 23657 raft_consensus.cc:2932] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } } }
I20240502 01:58:16.193717 22079 raft_consensus.cc:2932] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } } }
I20240502 01:58:16.195580 20809 catalog_manager.cc:5484] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:16.314323 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d024eee194347779dd13f423cc9023c"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59260
I20240502 01:58:16.314460 20950 raft_consensus.cc:595] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:16.524379 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d024eee194347779dd13f423cc9023c"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59278
I20240502 01:58:16.524502 20950 raft_consensus.cc:595] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:16.690093 23624 raft_consensus.cc:979] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: : Instructing follower 8690f48d41224e379f2c6c0f25251d4b to start an election
I20240502 01:58:16.690198 23624 raft_consensus.cc:1067] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer 8690f48d41224e379f2c6c0f25251d4b to start an election
I20240502 01:58:16.690620 22079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d024eee194347779dd13f423cc9023c"
dest_uuid: "8690f48d41224e379f2c6c0f25251d4b"
from {username='slave'} at 127.17.243.1:37839
I20240502 01:58:16.690748 22079 raft_consensus.cc:482] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:16.690809 22079 raft_consensus.cc:3037] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:16.691592 22079 raft_consensus.cc:504] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:16.691874 22079 leader_election.cc:288] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:16.692355 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024eee194347779dd13f423cc9023c" candidate_uuid: "8690f48d41224e379f2c6c0f25251d4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:16.692484 20950 raft_consensus.cc:3032] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:16.692531 20950 raft_consensus.cc:729] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:16.692668 20950 consensus_queue.cc:260] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:16.692795 20950 raft_consensus.cc:3037] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:16.693550 20950 raft_consensus.cc:2454] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8690f48d41224e379f2c6c0f25251d4b in term 2.
I20240502 01:58:16.693832 22013 leader_election.cc:302] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, 8690f48d41224e379f2c6c0f25251d4b; no voters:
I20240502 01:58:16.694010 23505 raft_consensus.cc:2781] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:16.694145 23505 raft_consensus.cc:686] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 2 LEADER]: Becoming Leader. State: Replica: 8690f48d41224e379f2c6c0f25251d4b, State: Running, Role: LEADER
I20240502 01:58:16.694255 23505 consensus_queue.cc:237] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:16.695462 20809 catalog_manager.cc:5484] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3). New cstate: current_term: 2 leader_uuid: "8690f48d41224e379f2c6c0f25251d4b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:16.697819 20798 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 3d024eee194347779dd13f423cc9023c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 8690f48d41224e379f2c6c0f25251d4b (127.17.243.3:35731) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:58:17.163234 20950 raft_consensus.cc:1261] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer 8690f48d41224e379f2c6c0f25251d4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:17.163643 23505 consensus_queue.cc:1035] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:17.297673 22079 consensus_queue.cc:237] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } }
I20240502 01:58:17.298406 23505 raft_consensus.cc:2932] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } } }
I20240502 01:58:17.299757 20798 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d024eee194347779dd13f423cc9023c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:17.300691 20811 catalog_manager.cc:5484] T 3d024eee194347779dd13f423cc9023c P 8690f48d41224e379f2c6c0f25251d4b reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "8690f48d41224e379f2c6c0f25251d4b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8690f48d41224e379f2c6c0f25251d4b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 35731 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:17.303392 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d024eee194347779dd13f423cc9023c with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:17.303701 23715 tablet_replica.cc:330] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:17.303807 23715 raft_consensus.cc:2229] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:17.303891 23715 raft_consensus.cc:2258] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:17.304414 23715 ts_tablet_manager.cc:1875] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:17.306123 23715 ts_tablet_manager.cc:1888] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:17.306190 23715 log.cc:1196] T 3d024eee194347779dd13f423cc9023c P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d024eee194347779dd13f423cc9023c
I20240502 01:58:17.306612 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet 3d024eee194347779dd13f423cc9023c (table rebalance_test_table_2 [id=e5eb5c4bfbb140ec9d8781f1ca346fab]) successfully deleted
I20240502 01:58:17.399772 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.401235 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.401440 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.403998 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.416682 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.428314 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.431236 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.497545 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.499857 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.500963 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.504317 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.504415 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.513723 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.521030 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.729816 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.730873 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.731065 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.732823 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.733390 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.742445 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.747087 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.855470 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.856634 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.858011 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.858994 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.864526 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.872116 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.878849 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.968588 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.970348 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.970680 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:17.975133 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.975837 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.975986 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:17.977102 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
93e5e9f48cd5489d9256ca43be967240 | 127.17.243.62:43303 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.17.243.20:34307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3783e79423214efcba54d2bb50aa202f | 127.17.243.1:46147 | HEALTHY | <none> | 3 | 0
580cdba303094f18bc3a325acb7095ed | 127.17.243.2:36375 | HEALTHY | <none> | 3 | 0
63b02225f2b444f29dcf0041105f4784 | 127.17.243.5:38921 | HEALTHY | <none> | 2 | 0
8690f48d41224e379f2c6c0f25251d4b | 127.17.243.3:35731 | HEALTHY | <none> | 2 | 0
87578246c04b49188edc94b74619a22c | 127.17.243.4:44049 | HEALTHY | <none> | 2 | 0
998441e608064fad81a173c929277899 | 127.17.243.7:33251 | HEALTHY | <none> | 2 | 0
c1449418fd3344ce996bf17e7bade3b7 | 127.17.243.6:44947 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.17.243.1 | experimental | 127.17.243.1:46147
local_ip_for_outbound_sockets | 127.17.243.2 | experimental | 127.17.243.2:36375
local_ip_for_outbound_sockets | 127.17.243.3 | experimental | 127.17.243.3:35731
local_ip_for_outbound_sockets | 127.17.243.4 | experimental | 127.17.243.4:44049
local_ip_for_outbound_sockets | 127.17.243.5 | experimental | 127.17.243.5:38921
local_ip_for_outbound_sockets | 127.17.243.6 | experimental | 127.17.243.6:44947
local_ip_for_outbound_sockets | 127.17.243.7 | experimental | 127.17.243.7:33251
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.17.243.1:46147
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.17.243.2:36375
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.17.243.3:35731
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.17.243.4:44049
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.17.243.5:38921
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.17.243.6:44947
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.17.243.7:33251
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.17.243.20:34307 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:58:18.021530 18380 log_verifier.cc:126] Checking tablet 14f887acbed24c2fac257df46bcc3a8c
I20240502 01:58:18.032058 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 14f887acbed24c2fac257df46bcc3a8c
I20240502 01:58:18.032128 18380 log_verifier.cc:126] Checking tablet 194f70e09ac6452fbc18ef8481334607
I20240502 01:58:18.039922 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 194f70e09ac6452fbc18ef8481334607
I20240502 01:58:18.039968 18380 log_verifier.cc:126] Checking tablet 3a5c72f9e6e34cae985bce2cab15cfd4
I20240502 01:58:18.048496 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3a5c72f9e6e34cae985bce2cab15cfd4
I20240502 01:58:18.048548 18380 log_verifier.cc:126] Checking tablet 3d024eee194347779dd13f423cc9023c
I20240502 01:58:18.056990 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3d024eee194347779dd13f423cc9023c
I20240502 01:58:18.057034 18380 log_verifier.cc:126] Checking tablet 56b2393c70cb4dcfbc7e8ba06e67124b
I20240502 01:58:18.065416 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 56b2393c70cb4dcfbc7e8ba06e67124b
I20240502 01:58:18.065462 18380 log_verifier.cc:126] Checking tablet 5fbd475017a14bf8b488f2830e2bcee9
I20240502 01:58:18.073704 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5fbd475017a14bf8b488f2830e2bcee9
I20240502 01:58:18.073755 18380 log_verifier.cc:126] Checking tablet 6d31ece9e2b14c35957270ffd6a7b33f
I20240502 01:58:18.081812 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6d31ece9e2b14c35957270ffd6a7b33f
I20240502 01:58:18.081857 18380 log_verifier.cc:126] Checking tablet 79511d04d45d4c00821f7bcc3231c1a9
I20240502 01:58:18.090360 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 79511d04d45d4c00821f7bcc3231c1a9
I20240502 01:58:18.090402 18380 log_verifier.cc:126] Checking tablet 795c3440189a488ea97073a9f2e5856f
I20240502 01:58:18.098903 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 795c3440189a488ea97073a9f2e5856f
I20240502 01:58:18.098948 18380 log_verifier.cc:126] Checking tablet 92bd8d25f42446dc8c832b9ca46f17c8
I20240502 01:58:18.106920 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92bd8d25f42446dc8c832b9ca46f17c8
I20240502 01:58:18.106962 18380 log_verifier.cc:126] Checking tablet a8c274b327e04a85b39ef8984a6f486f
I20240502 01:58:18.115207 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a8c274b327e04a85b39ef8984a6f486f
I20240502 01:58:18.115264 18380 log_verifier.cc:126] Checking tablet b164633a362e4318975b74a1c5ea362e
I20240502 01:58:18.123430 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b164633a362e4318975b74a1c5ea362e
I20240502 01:58:18.123478 18380 log_verifier.cc:126] Checking tablet c13461f24fc94b938cd2e524dd8a9df2
I20240502 01:58:18.130517 18380 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c13461f24fc94b938cd2e524dd8a9df2
I20240502 01:58:18.130555 18380 log_verifier.cc:126] Checking tablet c4fd077b7bcc43478ba93f21de746397
I20240502 01:58:18.138484 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c4fd077b7bcc43478ba93f21de746397
I20240502 01:58:18.138523 18380 log_verifier.cc:126] Checking tablet c88b79b757d8428a89debee2187ed755
I20240502 01:58:18.146463 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c88b79b757d8428a89debee2187ed755
I20240502 01:58:18.146503 18380 log_verifier.cc:126] Checking tablet cad36bda76a24e3da0bcd937b30d48a1
I20240502 01:58:18.156340 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet cad36bda76a24e3da0bcd937b30d48a1
I20240502 01:58:18.158402 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.8:0
--local_ip_for_outbound_sockets=127.17.243.8
--webserver_interface=127.17.243.8
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--builtin_ntp_servers=127.17.243.20:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:18.259505 23898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:18.259677 23898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:18.259730 23898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:18.262637 23898 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:18.262691 23898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:18.262768 23898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.8
I20240502 01:58:18.266230 23898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.243.8
--webserver_port=0
--tserver_master_addrs=127.17.243.62:43303
--never_fsync=true
--heap_profile_path=/tmp/kudu.23898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.8
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:18.266963 23898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:18.267936 23898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:18.275815 23898 server_base.cc:1034] running on GCE node
W20240502 01:58:18.276136 23903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:18.276613 23904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:18.277043 23906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:18.277384 23898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:18.277864 23898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:18.279001 23898 hybrid_clock.cc:648] HybridClock initialized: now 1714615098278978 us; error 35 us; skew 500 ppm
I20240502 01:58:18.280536 23898 webserver.cc:460] Webserver started at http://127.17.243.8:45689/ using document root <none> and password file <none>
I20240502 01:58:18.281005 23898 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:18.281059 23898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:18.281213 23898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:18.282733 23898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "bd2f85b725fa4d3e99995ede99a7bd89"
format_stamp: "Formatted at 2024-05-02 01:58:18 on dist-test-slave-j3s5"
I20240502 01:58:18.283154 23898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "bd2f85b725fa4d3e99995ede99a7bd89"
format_stamp: "Formatted at 2024-05-02 01:58:18 on dist-test-slave-j3s5"
I20240502 01:58:18.286312 23898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:18.288323 23912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:18.289244 23898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:18.289337 23898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bd2f85b725fa4d3e99995ede99a7bd89"
format_stamp: "Formatted at 2024-05-02 01:58:18 on dist-test-slave-j3s5"
I20240502 01:58:18.289412 23898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:18.306720 23898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:18.307236 23898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:18.307370 23898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:18.307822 23898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:18.308591 23898 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:18.308636 23898 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:18.308683 23898 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:18.308712 23898 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:18.317654 23898 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.8:39189
I20240502 01:58:18.317701 24025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.8:39189 every 8 connection(s)
I20240502 01:58:18.318536 23898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:58:18.323290 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 23898
I20240502 01:58:18.323413 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:58:18.326953 24026 heartbeater.cc:347] Connected to a master server at 127.17.243.62:43303
I20240502 01:58:18.327164 24026 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:18.327740 24026 heartbeater.cc:511] Master 127.17.243.62:43303 requested a full tablet report, sending...
I20240502 01:58:18.328819 20810 ts_manager.cc:194] Registered new tserver with Master: bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8:39189)
I20240502 01:58:18.329540 20810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.8:32885
I20240502 01:58:18.506916 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:18.508383 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.510063 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.516888 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.518218 23960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:18.519290 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.526602 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:18.534945 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.646294 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.648021 20933 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:18.649609 21063 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:18.651592 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.651839 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.653191 23960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:18.669044 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.671386 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:18.721912 21084 consensus_queue.cc:237] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } }
I20240502 01:58:18.722973 24119 raft_consensus.cc:2932] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } } }
I20240502 01:58:18.725826 20802 catalog_manager.cc:5484] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index -1 to 4, NON_VOTER bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:18.727613 21018 consensus_peers.cc:485] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed -> Peer bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8:39189): Couldn't send request to peer bd2f85b725fa4d3e99995ede99a7bd89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 194f70e09ac6452fbc18ef8481334607. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:18.733381 20950 consensus_queue.cc:237] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } }
I20240502 01:58:18.734261 24132 raft_consensus.cc:2932] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } } }
I20240502 01:58:18.736582 20802 catalog_manager.cc:5484] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index -1 to 4, NON_VOTER bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:18.739035 20886 consensus_peers.cc:485] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f -> Peer bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8:39189): Couldn't send request to peer bd2f85b725fa4d3e99995ede99a7bd89. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d31ece9e2b14c35957270ffd6a7b33f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:19.188076 24165 ts_tablet_manager.cc:902] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Initiating tablet copy from peer 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:19.188717 24165 tablet_copy_client.cc:308] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:36375
I20240502 01:58:19.191905 21105 tablet_copy_service.cc:140] P 580cdba303094f18bc3a325acb7095ed: Received BeginTabletCopySession request for tablet 194f70e09ac6452fbc18ef8481334607 from peer bd2f85b725fa4d3e99995ede99a7bd89 ({username='slave'} at 127.17.243.8:54851)
I20240502 01:58:19.192003 21105 tablet_copy_service.cc:161] P 580cdba303094f18bc3a325acb7095ed: Beginning new tablet copy session on tablet 194f70e09ac6452fbc18ef8481334607 from peer bd2f85b725fa4d3e99995ede99a7bd89 at {username='slave'} at 127.17.243.8:54851: session id = bd2f85b725fa4d3e99995ede99a7bd89-194f70e09ac6452fbc18ef8481334607
I20240502 01:58:19.192869 21105 tablet_copy_source_session.cc:205] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:19.195705 24165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194f70e09ac6452fbc18ef8481334607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:19.200081 24165 tablet_copy_client.cc:783] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:19.200242 24165 tablet_copy_client.cc:647] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:19.201560 24165 tablet_copy_client.cc:523] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:19.202926 24165 tablet_bootstrap.cc:492] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Bootstrap starting.
I20240502 01:58:19.207561 24165 log.cc:824] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:19.225080 24165 tablet_bootstrap.cc:492] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:19.225337 24165 tablet_bootstrap.cc:492] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Bootstrap complete.
I20240502 01:58:19.225582 24165 ts_tablet_manager.cc:1367] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.004s
I20240502 01:58:19.227334 24165 raft_consensus.cc:348] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } }
I20240502 01:58:19.227535 24165 raft_consensus.cc:729] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd2f85b725fa4d3e99995ede99a7bd89, State: Initialized, Role: LEARNER
I20240502 01:58:19.227820 24165 consensus_queue.cc:260] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } }
I20240502 01:58:19.228389 24026 heartbeater.cc:503] Master 127.17.243.62:43303 was elected leader, sending a full tablet report...
I20240502 01:58:19.228600 24165 ts_tablet_manager.cc:1398] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240502 01:58:19.229220 21105 tablet_copy_service.cc:342] P 580cdba303094f18bc3a325acb7095ed: Request end of tablet copy session bd2f85b725fa4d3e99995ede99a7bd89-194f70e09ac6452fbc18ef8481334607 received from {username='slave'} at 127.17.243.8:54851
I20240502 01:58:19.229312 21105 tablet_copy_service.cc:433] P 580cdba303094f18bc3a325acb7095ed: ending tablet copy session bd2f85b725fa4d3e99995ede99a7bd89-194f70e09ac6452fbc18ef8481334607 on tablet 194f70e09ac6452fbc18ef8481334607 with peer bd2f85b725fa4d3e99995ede99a7bd89
I20240502 01:58:19.268265 24165 ts_tablet_manager.cc:902] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: Initiating tablet copy from peer 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:19.268656 24165 tablet_copy_client.cc:308] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:46147
I20240502 01:58:19.271639 20974 tablet_copy_service.cc:140] P 3783e79423214efcba54d2bb50aa202f: Received BeginTabletCopySession request for tablet 6d31ece9e2b14c35957270ffd6a7b33f from peer bd2f85b725fa4d3e99995ede99a7bd89 ({username='slave'} at 127.17.243.8:49211)
I20240502 01:58:19.271732 20974 tablet_copy_service.cc:161] P 3783e79423214efcba54d2bb50aa202f: Beginning new tablet copy session on tablet 6d31ece9e2b14c35957270ffd6a7b33f from peer bd2f85b725fa4d3e99995ede99a7bd89 at {username='slave'} at 127.17.243.8:49211: session id = bd2f85b725fa4d3e99995ede99a7bd89-6d31ece9e2b14c35957270ffd6a7b33f
I20240502 01:58:19.272379 20974 tablet_copy_source_session.cc:205] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:19.273108 24165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d31ece9e2b14c35957270ffd6a7b33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:19.274652 24165 tablet_copy_client.cc:783] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:19.274796 24165 tablet_copy_client.cc:647] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:19.275786 24165 tablet_copy_client.cc:523] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:19.276628 24165 tablet_bootstrap.cc:492] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: Bootstrap starting.
I20240502 01:58:19.293035 24165 tablet_bootstrap.cc:492] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:19.293236 24165 tablet_bootstrap.cc:492] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: Bootstrap complete.
I20240502 01:58:19.293372 24165 ts_tablet_manager.cc:1367] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240502 01:58:19.293810 24165 raft_consensus.cc:348] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } }
I20240502 01:58:19.293920 24165 raft_consensus.cc:729] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd2f85b725fa4d3e99995ede99a7bd89, State: Initialized, Role: LEARNER
I20240502 01:58:19.294143 24165 consensus_queue.cc:260] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: true } }
I20240502 01:58:19.294456 24165 ts_tablet_manager.cc:1398] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 01:58:19.294907 20974 tablet_copy_service.cc:342] P 3783e79423214efcba54d2bb50aa202f: Request end of tablet copy session bd2f85b725fa4d3e99995ede99a7bd89-6d31ece9e2b14c35957270ffd6a7b33f received from {username='slave'} at 127.17.243.8:49211
I20240502 01:58:19.294986 20974 tablet_copy_service.cc:433] P 3783e79423214efcba54d2bb50aa202f: ending tablet copy session bd2f85b725fa4d3e99995ede99a7bd89-6d31ece9e2b14c35957270ffd6a7b33f on tablet 6d31ece9e2b14c35957270ffd6a7b33f with peer bd2f85b725fa4d3e99995ede99a7bd89
I20240502 01:58:19.640798 23980 raft_consensus.cc:1203] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:19.745740 23980 raft_consensus.cc:1203] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:20.206519 24223 raft_consensus.cc:1050] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: attempting to promote NON_VOTER bd2f85b725fa4d3e99995ede99a7bd89 to VOTER
I20240502 01:58:20.206866 24223 consensus_queue.cc:237] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.207957 23980 raft_consensus.cc:1261] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Refusing update from remote peer 580cdba303094f18bc3a325acb7095ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:20.208369 24223 consensus_queue.cc:1035] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:20.211196 24224 raft_consensus.cc:2932] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } } }
I20240502 01:58:20.211688 24202 raft_consensus.cc:1050] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: attempting to promote NON_VOTER bd2f85b725fa4d3e99995ede99a7bd89 to VOTER
I20240502 01:58:20.212035 24202 consensus_queue.cc:237] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.211807 23980 raft_consensus.cc:2932] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } } }
I20240502 01:58:20.213070 23979 raft_consensus.cc:1261] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 LEARNER]: Refusing update from remote peer 3783e79423214efcba54d2bb50aa202f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:20.213892 24202 consensus_queue.cc:1035] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:20.214016 20802 catalog_manager.cc:5484] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed reported cstate change: config changed from index 4 to 5, bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "580cdba303094f18bc3a325acb7095ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:20.215539 24201 raft_consensus.cc:2932] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } } }
I20240502 01:58:20.216038 23980 raft_consensus.cc:2932] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } } }
I20240502 01:58:20.217864 20803 catalog_manager.cc:5484] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f reported cstate change: config changed from index 4 to 5, bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3783e79423214efcba54d2bb50aa202f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:20.229557 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d31ece9e2b14c35957270ffd6a7b33f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50474
I20240502 01:58:20.229660 20950 raft_consensus.cc:595] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:20.238376 21084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "194f70e09ac6452fbc18ef8481334607"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51912
I20240502 01:58:20.238492 21084 raft_consensus.cc:595] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:20.449126 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d31ece9e2b14c35957270ffd6a7b33f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50502
I20240502 01:58:20.449265 20950 raft_consensus.cc:595] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:20.458560 21084 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "194f70e09ac6452fbc18ef8481334607"
dest_uuid: "580cdba303094f18bc3a325acb7095ed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51934
I20240502 01:58:20.458678 21084 raft_consensus.cc:595] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:20.597723 24224 raft_consensus.cc:979] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: : Instructing follower bd2f85b725fa4d3e99995ede99a7bd89 to start an election
I20240502 01:58:20.597831 24224 raft_consensus.cc:1067] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Signalling peer bd2f85b725fa4d3e99995ede99a7bd89 to start an election
I20240502 01:58:20.598335 23980 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "194f70e09ac6452fbc18ef8481334607"
dest_uuid: "bd2f85b725fa4d3e99995ede99a7bd89"
from {username='slave'} at 127.17.243.2:52035
I20240502 01:58:20.598516 23980 raft_consensus.cc:482] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:20.598596 23980 raft_consensus.cc:3037] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:20.599360 23980 raft_consensus.cc:504] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.600189 23980 leader_election.cc:288] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [CANDIDATE]: Term 2 election: Requested vote from peers 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375)
I20240502 01:58:20.600548 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194f70e09ac6452fbc18ef8481334607" candidate_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "580cdba303094f18bc3a325acb7095ed"
I20240502 01:58:20.600661 21084 raft_consensus.cc:3032] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:20.600706 21084 raft_consensus.cc:729] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 580cdba303094f18bc3a325acb7095ed, State: Running, Role: LEADER
I20240502 01:58:20.600843 21084 consensus_queue.cc:260] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.600972 21084 raft_consensus.cc:3037] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:20.601691 21084 raft_consensus.cc:2454] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2f85b725fa4d3e99995ede99a7bd89 in term 2.
I20240502 01:58:20.602067 23914 leader_election.cc:302] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 580cdba303094f18bc3a325acb7095ed, bd2f85b725fa4d3e99995ede99a7bd89; no voters:
I20240502 01:58:20.602277 24172 raft_consensus.cc:2781] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:20.602455 24172 raft_consensus.cc:686] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 LEADER]: Becoming Leader. State: Replica: bd2f85b725fa4d3e99995ede99a7bd89, State: Running, Role: LEADER
I20240502 01:58:20.602703 24172 consensus_queue.cc:237] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.604439 20803 catalog_manager.cc:5484] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 reported cstate change: term changed from 1 to 2, leader changed from 580cdba303094f18bc3a325acb7095ed (127.17.243.2) to bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8). New cstate: current_term: 2 leader_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:20.609750 20795 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 194f70e09ac6452fbc18ef8481334607 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8:39189) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:58:20.668857 20950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d31ece9e2b14c35957270ffd6a7b33f"
dest_uuid: "3783e79423214efcba54d2bb50aa202f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50518
I20240502 01:58:20.668991 20950 raft_consensus.cc:595] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:20.792623 24292 raft_consensus.cc:979] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: : Instructing follower bd2f85b725fa4d3e99995ede99a7bd89 to start an election
I20240502 01:58:20.792716 24292 raft_consensus.cc:1067] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Signalling peer bd2f85b725fa4d3e99995ede99a7bd89 to start an election
I20240502 01:58:20.793135 23980 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d31ece9e2b14c35957270ffd6a7b33f"
dest_uuid: "bd2f85b725fa4d3e99995ede99a7bd89"
from {username='slave'} at 127.17.243.1:33383
I20240502 01:58:20.793259 23980 raft_consensus.cc:482] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:20.793306 23980 raft_consensus.cc:3037] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:20.794080 23980 raft_consensus.cc:504] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.794401 23980 leader_election.cc:288] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [CANDIDATE]: Term 2 election: Requested vote from peers 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147)
I20240502 01:58:20.794759 20950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d31ece9e2b14c35957270ffd6a7b33f" candidate_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3783e79423214efcba54d2bb50aa202f"
I20240502 01:58:20.794878 20950 raft_consensus.cc:3032] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:20.794924 20950 raft_consensus.cc:729] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3783e79423214efcba54d2bb50aa202f, State: Running, Role: LEADER
I20240502 01:58:20.795063 20950 consensus_queue.cc:260] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.795187 20950 raft_consensus.cc:3037] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:20.795912 20950 raft_consensus.cc:2454] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2f85b725fa4d3e99995ede99a7bd89 in term 2.
I20240502 01:58:20.796212 23914 leader_election.cc:302] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3783e79423214efcba54d2bb50aa202f, bd2f85b725fa4d3e99995ede99a7bd89; no voters:
I20240502 01:58:20.796376 24172 raft_consensus.cc:2781] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:20.796474 24172 raft_consensus.cc:686] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 LEADER]: Becoming Leader. State: Replica: bd2f85b725fa4d3e99995ede99a7bd89, State: Running, Role: LEADER
I20240502 01:58:20.796599 24172 consensus_queue.cc:237] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:20.797868 20803 catalog_manager.cc:5484] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 reported cstate change: term changed from 1 to 2, leader changed from 3783e79423214efcba54d2bb50aa202f (127.17.243.1) to bd2f85b725fa4d3e99995ede99a7bd89 (127.17.243.8). New cstate: current_term: 2 leader_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:21.195453 21084 raft_consensus.cc:1261] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Refusing update from remote peer bd2f85b725fa4d3e99995ede99a7bd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:21.195956 24172 consensus_queue.cc:1035] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580cdba303094f18bc3a325acb7095ed" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 36375 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:21.209416 20950 raft_consensus.cc:1261] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Refusing update from remote peer bd2f85b725fa4d3e99995ede99a7bd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:21.209821 24172 consensus_queue.cc:1035] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3783e79423214efcba54d2bb50aa202f" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 46147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:21.231751 23980 consensus_queue.cc:237] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:21.232234 24172 raft_consensus.cc:2932] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } } }
I20240502 01:58:21.233341 20795 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 194f70e09ac6452fbc18ef8481334607 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:21.234066 20802 catalog_manager.cc:5484] T 194f70e09ac6452fbc18ef8481334607 P bd2f85b725fa4d3e99995ede99a7bd89 reported cstate change: config changed from index 5 to 7, VOTER 580cdba303094f18bc3a325acb7095ed (127.17.243.2) evicted. New cstate: current_term: 2 leader_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:21.236088 21063 tablet_service.cc:1514] Processing DeleteTablet for tablet 194f70e09ac6452fbc18ef8481334607 with delete_type TABLET_DATA_TOMBSTONED (TS 580cdba303094f18bc3a325acb7095ed not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50226
I20240502 01:58:21.236408 24317 tablet_replica.cc:330] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: stopping tablet replica
I20240502 01:58:21.236526 24317 raft_consensus.cc:2229] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:21.236621 24317 raft_consensus.cc:2258] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:21.237653 24317 ts_tablet_manager.cc:1875] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:21.239287 24317 ts_tablet_manager.cc:1888] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:21.239356 24317 log.cc:1196] T 194f70e09ac6452fbc18ef8481334607 P 580cdba303094f18bc3a325acb7095ed: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/194f70e09ac6452fbc18ef8481334607
I20240502 01:58:21.239780 20796 catalog_manager.cc:4830] TS 580cdba303094f18bc3a325acb7095ed (127.17.243.2:36375): tablet 194f70e09ac6452fbc18ef8481334607 (table rebalance_test_table_1 [id=5508a27a4396491786f2f6fd3b5a8632]) successfully deleted
I20240502 01:58:21.421250 23980 consensus_queue.cc:237] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } }
I20240502 01:58:21.421958 24172 raft_consensus.cc:2932] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } } }
I20240502 01:58:21.423218 20795 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d31ece9e2b14c35957270ffd6a7b33f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:58:21.424021 20802 catalog_manager.cc:5484] T 6d31ece9e2b14c35957270ffd6a7b33f P bd2f85b725fa4d3e99995ede99a7bd89 reported cstate change: config changed from index 5 to 7, VOTER 3783e79423214efcba54d2bb50aa202f (127.17.243.1) evicted. New cstate: current_term: 2 leader_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd2f85b725fa4d3e99995ede99a7bd89" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 39189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:21.426313 20933 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d31ece9e2b14c35957270ffd6a7b33f with delete_type TABLET_DATA_TOMBSTONED (TS 3783e79423214efcba54d2bb50aa202f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58528
I20240502 01:58:21.426662 24333 tablet_replica.cc:330] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: stopping tablet replica
I20240502 01:58:21.426777 24333 raft_consensus.cc:2229] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:21.426863 24333 raft_consensus.cc:2258] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:21.427876 24333 ts_tablet_manager.cc:1875] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:21.429570 24333 ts_tablet_manager.cc:1888] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:21.429634 24333 log.cc:1196] T 6d31ece9e2b14c35957270ffd6a7b33f P 3783e79423214efcba54d2bb50aa202f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d31ece9e2b14c35957270ffd6a7b33f
I20240502 01:58:21.430083 20796 catalog_manager.cc:4830] TS 3783e79423214efcba54d2bb50aa202f (127.17.243.1:46147): tablet 6d31ece9e2b14c35957270ffd6a7b33f (table rebalance_test_table_0 [id=b99bc3270af143db8e5641075113894b]) successfully deleted
I20240502 01:58:21.570574 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.570909 20933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.577728 21063 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.584900 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.585358 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.586303 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.586982 23960 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.608203 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.678709 20933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.686036 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.686847 21063 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.686959 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.687970 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.688136 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.695040 23960 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.709210 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.792207 21063 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.793804 20933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.797961 22321 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.800783 22059 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.808892 23960 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.811805 22190 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.819461 22452 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:21.826438 22584 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
93e5e9f48cd5489d9256ca43be967240 | 127.17.243.62:43303 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.17.243.20:34307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3783e79423214efcba54d2bb50aa202f | 127.17.243.1:46147 | HEALTHY | <none> | 2 | 0
580cdba303094f18bc3a325acb7095ed | 127.17.243.2:36375 | HEALTHY | <none> | 2 | 0
63b02225f2b444f29dcf0041105f4784 | 127.17.243.5:38921 | HEALTHY | <none> | 2 | 0
8690f48d41224e379f2c6c0f25251d4b | 127.17.243.3:35731 | HEALTHY | <none> | 2 | 0
87578246c04b49188edc94b74619a22c | 127.17.243.4:44049 | HEALTHY | <none> | 2 | 0
998441e608064fad81a173c929277899 | 127.17.243.7:33251 | HEALTHY | <none> | 2 | 0
bd2f85b725fa4d3e99995ede99a7bd89 | 127.17.243.8:39189 | HEALTHY | <none> | 2 | 0
c1449418fd3344ce996bf17e7bade3b7 | 127.17.243.6:44947 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.17.243.1 | experimental | 127.17.243.1:46147
local_ip_for_outbound_sockets | 127.17.243.2 | experimental | 127.17.243.2:36375
local_ip_for_outbound_sockets | 127.17.243.3 | experimental | 127.17.243.3:35731
local_ip_for_outbound_sockets | 127.17.243.4 | experimental | 127.17.243.4:44049
local_ip_for_outbound_sockets | 127.17.243.5 | experimental | 127.17.243.5:38921
local_ip_for_outbound_sockets | 127.17.243.6 | experimental | 127.17.243.6:44947
local_ip_for_outbound_sockets | 127.17.243.7 | experimental | 127.17.243.7:33251
local_ip_for_outbound_sockets | 127.17.243.8 | experimental | 127.17.243.8:39189
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.17.243.1:46147
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.17.243.2:36375
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.17.243.3:35731
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.17.243.4:44049
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.17.243.5:38921
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.17.243.6:44947
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.17.243.7:33251
server_dump_info_path | /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.17.243.8:39189
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.17.243.20:34307 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:58:21.849731 18380 log_verifier.cc:126] Checking tablet 14f887acbed24c2fac257df46bcc3a8c
I20240502 01:58:21.855648 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 14f887acbed24c2fac257df46bcc3a8c
I20240502 01:58:21.855688 18380 log_verifier.cc:126] Checking tablet 194f70e09ac6452fbc18ef8481334607
I20240502 01:58:21.863752 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 194f70e09ac6452fbc18ef8481334607
I20240502 01:58:21.863801 18380 log_verifier.cc:126] Checking tablet 3a5c72f9e6e34cae985bce2cab15cfd4
I20240502 01:58:21.869292 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3a5c72f9e6e34cae985bce2cab15cfd4
I20240502 01:58:21.869335 18380 log_verifier.cc:126] Checking tablet 3d024eee194347779dd13f423cc9023c
I20240502 01:58:21.874707 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3d024eee194347779dd13f423cc9023c
I20240502 01:58:21.874747 18380 log_verifier.cc:126] Checking tablet 56b2393c70cb4dcfbc7e8ba06e67124b
I20240502 01:58:21.880020 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 56b2393c70cb4dcfbc7e8ba06e67124b
I20240502 01:58:21.880057 18380 log_verifier.cc:126] Checking tablet 5fbd475017a14bf8b488f2830e2bcee9
I20240502 01:58:21.885352 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5fbd475017a14bf8b488f2830e2bcee9
I20240502 01:58:21.885390 18380 log_verifier.cc:126] Checking tablet 6d31ece9e2b14c35957270ffd6a7b33f
I20240502 01:58:21.893378 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6d31ece9e2b14c35957270ffd6a7b33f
I20240502 01:58:21.893419 18380 log_verifier.cc:126] Checking tablet 79511d04d45d4c00821f7bcc3231c1a9
I20240502 01:58:21.898752 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 79511d04d45d4c00821f7bcc3231c1a9
I20240502 01:58:21.898793 18380 log_verifier.cc:126] Checking tablet 795c3440189a488ea97073a9f2e5856f
I20240502 01:58:21.904094 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 795c3440189a488ea97073a9f2e5856f
I20240502 01:58:21.904134 18380 log_verifier.cc:126] Checking tablet 92bd8d25f42446dc8c832b9ca46f17c8
I20240502 01:58:21.909212 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92bd8d25f42446dc8c832b9ca46f17c8
I20240502 01:58:21.909250 18380 log_verifier.cc:126] Checking tablet a8c274b327e04a85b39ef8984a6f486f
I20240502 01:58:21.914649 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a8c274b327e04a85b39ef8984a6f486f
I20240502 01:58:21.914688 18380 log_verifier.cc:126] Checking tablet b164633a362e4318975b74a1c5ea362e
I20240502 01:58:21.919948 18380 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b164633a362e4318975b74a1c5ea362e
I20240502 01:58:21.919986 18380 log_verifier.cc:126] Checking tablet c13461f24fc94b938cd2e524dd8a9df2
I20240502 01:58:21.924211 18380 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c13461f24fc94b938cd2e524dd8a9df2
I20240502 01:58:21.924252 18380 log_verifier.cc:126] Checking tablet c4fd077b7bcc43478ba93f21de746397
I20240502 01:58:21.929594 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c4fd077b7bcc43478ba93f21de746397
I20240502 01:58:21.929633 18380 log_verifier.cc:126] Checking tablet c88b79b757d8428a89debee2187ed755
I20240502 01:58:21.935109 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c88b79b757d8428a89debee2187ed755
I20240502 01:58:21.935149 18380 log_verifier.cc:126] Checking tablet cad36bda76a24e3da0bcd937b30d48a1
I20240502 01:58:21.940477 18380 log_verifier.cc:177] Verified matching terms for 7 ops in tablet cad36bda76a24e3da0bcd937b30d48a1
I20240502 01:58:21.940872 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 20871
I20240502 01:58:21.947503 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21003
I20240502 01:58:21.953298 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 21997
I20240502 01:58:21.957897 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 22128
I20240502 01:58:21.962280 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 22259
I20240502 01:58:21.967098 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 22390
I20240502 01:58:21.971858 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 22521
I20240502 01:58:21.976114 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 23898
I20240502 01:58:21.980302 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 20780
2024-05-02T01:58:21Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (17828 ms)
[----------] 1 test from RebalanceParamTest (17828 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240502 01:58:22.013490 18380 test_util.cc:268] Using random seed: -267059153
I20240502 01:58:22.014189 18380 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:58:22.014221 18380 ts_itest-base.cc:113] --------------
I20240502 01:58:22.014231 18380 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:58:22.014240 18380 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:58:22.014251 18380 ts_itest-base.cc:116] --------------
2024-05-02T01:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:58:22Z Disabled control of system clock
I20240502 01:58:22.021462 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:33801
--webserver_interface=127.17.243.62
--webserver_port=0
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.243.62:33801
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:58:22.120296 24456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:22.120476 24456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:22.120515 24456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:22.123417 24456 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:22.123471 24456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:58:22.123497 24456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:22.123528 24456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:58:22.123556 24456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:58:22.127028 24456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.17.243.62:33801
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.243.62:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.243.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:22.127727 24456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:22.128726 24456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:22.135495 24462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.135905 24464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.136309 24461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:22.136700 24456 server_base.cc:1034] running on GCE node
I20240502 01:58:22.137136 24456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:22.137967 24456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:22.139120 24456 hybrid_clock.cc:648] HybridClock initialized: now 1714615102139097 us; error 33 us; skew 500 ppm
I20240502 01:58:22.140834 24456 webserver.cc:460] Webserver started at http://127.17.243.62:38789/ using document root <none> and password file <none>
I20240502 01:58:22.141320 24456 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:22.141376 24456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:22.141539 24456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:22.143039 24456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "52b027cea09c41dca3dade2a5ec516e9"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.143493 24456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "52b027cea09c41dca3dade2a5ec516e9"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.146572 24456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:22.149492 24470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.150414 24456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:22.150527 24456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
uuid: "52b027cea09c41dca3dade2a5ec516e9"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.150604 24456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:22.170850 24456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:22.171367 24456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:22.171502 24456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:22.177350 24456 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.62:33801
I20240502 01:58:22.177367 24523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.62:33801 every 8 connection(s)
I20240502 01:58:22.178191 24456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:58:22.180451 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:22.185690 24524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Bootstrap starting.
I20240502 01:58:22.185767 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 24456
I20240502 01:58:22.185868 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:58:22.187997 24524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:22.188731 24524 log.cc:824] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:22.190330 24524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: No bootstrap required, opened a new log
I20240502 01:58:22.193863 24524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b027cea09c41dca3dade2a5ec516e9" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33801 } }
I20240502 01:58:22.194087 24524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:22.194187 24524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b027cea09c41dca3dade2a5ec516e9, State: Initialized, Role: FOLLOWER
I20240502 01:58:22.194797 24524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b027cea09c41dca3dade2a5ec516e9" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33801 } }
I20240502 01:58:22.194953 24524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:22.195055 24524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:22.195168 24524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:22.195974 24524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b027cea09c41dca3dade2a5ec516e9" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33801 } }
I20240502 01:58:22.196445 24524 leader_election.cc:302] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52b027cea09c41dca3dade2a5ec516e9; no voters:
I20240502 01:58:22.196827 24524 leader_election.cc:288] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:22.197139 24529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:22.197358 24529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [term 1 LEADER]: Becoming Leader. State: Replica: 52b027cea09c41dca3dade2a5ec516e9, State: Running, Role: LEADER
I20240502 01:58:22.197639 24529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b027cea09c41dca3dade2a5ec516e9" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33801 } }
I20240502 01:58:22.197970 24524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:22.199151 24530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52b027cea09c41dca3dade2a5ec516e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b027cea09c41dca3dade2a5ec516e9" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33801 } } }
I20240502 01:58:22.199151 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52b027cea09c41dca3dade2a5ec516e9. Latest consensus state: current_term: 1 leader_uuid: "52b027cea09c41dca3dade2a5ec516e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b027cea09c41dca3dade2a5ec516e9" member_type: VOTER last_known_addr { host: "127.17.243.62" port: 33801 } } }
I20240502 01:58:22.199326 24530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:22.199405 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:22.200403 24540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:22.203356 24540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:22.208588 24540 catalog_manager.cc:1347] Generated new cluster ID: b47f78c7d5464932a0ebbfae48c45bcf
I20240502 01:58:22.208679 24540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:22.228775 24540 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:22.229724 24540 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:22.242419 24540 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Generated new TSK 0
I20240502 01:58:22.242969 24540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:22.249822 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.1:0
--local_ip_for_outbound_sockets=127.17.243.1
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:22.349551 24549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:22.349735 24549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:22.349794 24549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:22.352702 24549 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:22.352762 24549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:22.352840 24549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.1
I20240502 01:58:22.356374 24549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.243.1
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.24549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.1
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:22.357120 24549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:22.358155 24549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:22.365679 24554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.366022 24555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.366551 24557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:22.367012 24549 server_base.cc:1034] running on GCE node
I20240502 01:58:22.367342 24549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:22.367796 24549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:22.368940 24549 hybrid_clock.cc:648] HybridClock initialized: now 1714615102368925 us; error 27 us; skew 500 ppm
I20240502 01:58:22.370605 24549 webserver.cc:460] Webserver started at http://127.17.243.1:38367/ using document root <none> and password file <none>
I20240502 01:58:22.371074 24549 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:22.371140 24549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:22.371323 24549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:22.372792 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "588bfc090a5247598cd5d3cee5ad2007"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.373209 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "588bfc090a5247598cd5d3cee5ad2007"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.376101 24549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:22.378031 24563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.378896 24549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:22.379000 24549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "588bfc090a5247598cd5d3cee5ad2007"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.379072 24549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:22.403335 24549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:22.403797 24549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:22.403941 24549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:22.404372 24549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:22.405122 24549 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:22.405167 24549 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.405216 24549 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:22.405243 24549 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.412690 24549 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.1:36901
I20240502 01:58:22.412737 24677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.1:36901 every 8 connection(s)
I20240502 01:58:22.413486 24549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:58:22.414712 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 24549
I20240502 01:58:22.414803 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:58:22.416860 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.2:0
--local_ip_for_outbound_sockets=127.17.243.2
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:22.421980 24679 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:22.422194 24679 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:22.422753 24679 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:22.424265 24487 ts_manager.cc:194] Registered new tserver with Master: 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:22.425791 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.1:37273
W20240502 01:58:22.517510 24682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:22.517671 24682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:22.517711 24682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:22.520532 24682 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:22.520583 24682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:22.520643 24682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.2
I20240502 01:58:22.523948 24682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.243.2
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.24682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.2
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:22.524596 24682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:22.525503 24682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:22.532639 24689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.532960 24691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.533542 24687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:22.534122 24682 server_base.cc:1034] running on GCE node
I20240502 01:58:22.534435 24682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:22.534889 24682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:22.536031 24682 hybrid_clock.cc:648] HybridClock initialized: now 1714615102536023 us; error 22 us; skew 500 ppm
I20240502 01:58:22.537566 24682 webserver.cc:460] Webserver started at http://127.17.243.2:43593/ using document root <none> and password file <none>
I20240502 01:58:22.538061 24682 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:22.538116 24682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:22.538265 24682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:22.539686 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "92cb785c165749eca49af5784756461c"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.540094 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "92cb785c165749eca49af5784756461c"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.542945 24682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 01:58:22.544791 24697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.545660 24682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:22.545759 24682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "92cb785c165749eca49af5784756461c"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.545828 24682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:22.557245 24682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:22.557725 24682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:22.557865 24682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:22.558324 24682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:22.559059 24682 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:22.559105 24682 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.559151 24682 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:22.559180 24682 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.566669 24682 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.2:45237
I20240502 01:58:22.566696 24810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.2:45237 every 8 connection(s)
I20240502 01:58:22.567385 24682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:58:22.571761 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 24682
I20240502 01:58:22.571854 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:58:22.573889 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.3:0
--local_ip_for_outbound_sockets=127.17.243.3
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:22.575259 24811 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:22.575484 24811 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:22.576011 24811 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:22.576929 24487 ts_manager.cc:194] Registered new tserver with Master: 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:22.577639 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.2:37837
W20240502 01:58:22.674345 24814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:22.674511 24814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:22.674551 24814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:22.677354 24814 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:22.677404 24814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:22.677476 24814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.3
I20240502 01:58:22.680822 24814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.243.3
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.24814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.3
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:22.681484 24814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:22.682440 24814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:22.689949 24819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.690397 24820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.690812 24822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:22.691227 24814 server_base.cc:1034] running on GCE node
I20240502 01:58:22.691587 24814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:22.692090 24814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:22.693243 24814 hybrid_clock.cc:648] HybridClock initialized: now 1714615102693220 us; error 35 us; skew 500 ppm
I20240502 01:58:22.694938 24814 webserver.cc:460] Webserver started at http://127.17.243.3:46739/ using document root <none> and password file <none>
I20240502 01:58:22.695427 24814 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:22.695487 24814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:22.695672 24814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:22.697160 24814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f46bcd2bc124471981349b1e2726253b"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.697625 24814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f46bcd2bc124471981349b1e2726253b"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.700922 24814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:22.702898 24829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.703787 24814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:22.703883 24814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f46bcd2bc124471981349b1e2726253b"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.703966 24814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:22.724951 24814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:22.725409 24814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:22.725557 24814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:22.726028 24814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:22.726796 24814 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:22.726845 24814 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.726895 24814 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:22.726928 24814 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.735153 24814 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.3:45445
I20240502 01:58:22.735173 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.3:45445 every 8 connection(s)
I20240502 01:58:22.735925 24814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:58:22.739391 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 24814
I20240502 01:58:22.739502 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:58:22.741719 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.4:0
--local_ip_for_outbound_sockets=127.17.243.4
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:22.744745 24945 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:22.744944 24945 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:22.745397 24945 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:22.746249 24487 ts_manager.cc:194] Registered new tserver with Master: f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:22.746754 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.3:41507
W20240502 01:58:22.841709 24948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:22.841888 24948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:22.841971 24948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:22.844842 24948 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:22.844902 24948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:22.845001 24948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.4
I20240502 01:58:22.848479 24948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.243.4
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.24948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.4
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:22.849170 24948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:22.850162 24948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:22.857578 24954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.858040 24953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:22.858398 24956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:22.858824 24948 server_base.cc:1034] running on GCE node
I20240502 01:58:22.859134 24948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:22.859586 24948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:22.860733 24948 hybrid_clock.cc:648] HybridClock initialized: now 1714615102860707 us; error 40 us; skew 500 ppm
I20240502 01:58:22.862349 24948 webserver.cc:460] Webserver started at http://127.17.243.4:44581/ using document root <none> and password file <none>
I20240502 01:58:22.862814 24948 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:22.862866 24948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:22.863032 24948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:22.864440 24948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.864849 24948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.867734 24948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:22.869529 24962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.870402 24948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:22.870496 24948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
format_stamp: "Formatted at 2024-05-02 01:58:22 on dist-test-slave-j3s5"
I20240502 01:58:22.870568 24948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:22.887244 24948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:22.887671 24948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:22.887801 24948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:22.888223 24948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:22.888942 24948 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:22.888999 24948 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.889045 24948 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:22.889071 24948 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:22.896711 24948 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.4:42879
I20240502 01:58:22.896739 25079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.4:42879 every 8 connection(s)
I20240502 01:58:22.897475 24948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:22.905102 25080 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:22.905270 25080 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:22.905689 25080 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:22.906544 24487 ts_manager.cc:194] Registered new tserver with Master: e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:22.907127 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 24948
I20240502 01:58:22.907159 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.4:37869
I20240502 01:58:22.907235 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:58:22.909065 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.5:0
--local_ip_for_outbound_sockets=127.17.243.5
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:23.008066 25083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:23.008255 25083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:23.008302 25083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:23.011191 25083 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:23.011245 25083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:23.011322 25083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.5
I20240502 01:58:23.014729 25083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.243.5
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.25083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.5
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:23.015421 25083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:23.016350 25083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:23.023658 25088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.024097 25091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.024427 25089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:23.024863 25083 server_base.cc:1034] running on GCE node
I20240502 01:58:23.025207 25083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:23.025666 25083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:23.026816 25083 hybrid_clock.cc:648] HybridClock initialized: now 1714615103026808 us; error 25 us; skew 500 ppm
I20240502 01:58:23.028431 25083 webserver.cc:460] Webserver started at http://127.17.243.5:42201/ using document root <none> and password file <none>
I20240502 01:58:23.028918 25083 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:23.028972 25083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:23.029126 25083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:23.030612 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.031026 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.033882 25083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:23.035810 25097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.036669 25083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:23.036772 25083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.036844 25083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:23.056174 25083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:23.056633 25083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:23.056764 25083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:23.057188 25083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:23.057972 25083 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:23.058017 25083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.058063 25083 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:23.058090 25083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.066071 25083 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.5:40239
I20240502 01:58:23.066107 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.5:40239 every 8 connection(s)
I20240502 01:58:23.066836 25083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:23.073617 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 25083
I20240502 01:58:23.073714 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:58:23.074329 25213 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:23.074522 25213 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:23.075043 25213 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:23.075701 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.6:0
--local_ip_for_outbound_sockets=127.17.243.6
--webserver_interface=127.17.243.6
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:23.075886 24487 ts_manager.cc:194] Registered new tserver with Master: 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:23.076401 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.5:48051
W20240502 01:58:23.172974 25216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:23.173149 25216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:23.173187 25216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:23.176136 25216 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:23.176193 25216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:23.176270 25216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.6
I20240502 01:58:23.179630 25216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.243.6
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.25216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.6
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:23.180315 25216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:23.181244 25216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:23.188740 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.189266 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.189570 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:23.190064 25216 server_base.cc:1034] running on GCE node
I20240502 01:58:23.190385 25216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:23.190830 25216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:23.192003 25216 hybrid_clock.cc:648] HybridClock initialized: now 1714615103191984 us; error 26 us; skew 500 ppm
I20240502 01:58:23.193784 25216 webserver.cc:460] Webserver started at http://127.17.243.6:40299/ using document root <none> and password file <none>
I20240502 01:58:23.194334 25216 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:23.194396 25216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:23.194586 25216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:23.196089 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "1049b4c8783b48f39be22bd113ad349f"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.196537 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "1049b4c8783b48f39be22bd113ad349f"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.199833 25216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:23.201857 25230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.202769 25216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:23.202883 25216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1049b4c8783b48f39be22bd113ad349f"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.202963 25216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:23.217625 25216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:23.218120 25216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:23.218266 25216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:23.218725 25216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:23.219512 25216 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:23.219563 25216 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.219614 25216 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:23.219645 25216 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.227927 25216 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.6:41707
I20240502 01:58:23.227974 25344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.6:41707 every 8 connection(s)
I20240502 01:58:23.228713 25216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:23.230409 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 25216
I20240502 01:58:23.230542 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:58:23.232645 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.7:0
--local_ip_for_outbound_sockets=127.17.243.7
--webserver_interface=127.17.243.7
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:23.236797 25345 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:23.237002 25345 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:23.237459 25345 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:23.238404 24487 ts_manager.cc:194] Registered new tserver with Master: 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:23.239121 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.6:42495
W20240502 01:58:23.331733 25348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:23.331897 25348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:23.331936 25348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:23.334852 25348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:23.334904 25348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:23.334965 25348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.7
I20240502 01:58:23.338312 25348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.243.7
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.25348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.7
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:23.339006 25348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:23.339980 25348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:23.347491 25354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.347880 25353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.348290 25356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:23.348819 25348 server_base.cc:1034] running on GCE node
I20240502 01:58:23.349119 25348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:23.349588 25348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:23.350720 25348 hybrid_clock.cc:648] HybridClock initialized: now 1714615103350691 us; error 65 us; skew 500 ppm
I20240502 01:58:23.352340 25348 webserver.cc:460] Webserver started at http://127.17.243.7:39385/ using document root <none> and password file <none>
I20240502 01:58:23.352806 25348 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:23.352857 25348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:23.353024 25348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:23.354542 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.354974 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.357796 25348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:23.359690 25362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.360559 25348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:23.360656 25348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.360730 25348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:23.374035 25348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:23.374456 25348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:23.374574 25348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:23.374986 25348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:23.375713 25348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:23.375757 25348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.375793 25348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:23.375820 25348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.383344 25348 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.7:42021
I20240502 01:58:23.383369 25476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.7:42021 every 8 connection(s)
I20240502 01:58:23.384088 25348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:23.387235 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 25348
I20240502 01:58:23.387341 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:58:23.389292 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.8:0
--local_ip_for_outbound_sockets=127.17.243.8
--webserver_interface=127.17.243.8
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:23.391880 25477 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:23.392062 25477 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:23.392480 25477 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:23.393342 24487 ts_manager.cc:194] Registered new tserver with Master: 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:23.393839 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.7:41169
I20240502 01:58:23.428346 24679 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
W20240502 01:58:23.488763 25480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:23.488924 25480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:23.488981 25480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:23.491847 25480 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:23.491904 25480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:23.491982 25480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.8
I20240502 01:58:23.495317 25480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.243.8
--webserver_port=0
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.25480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.8
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:23.495982 25480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:23.496861 25480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:23.504040 25485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.504354 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:23.504688 25488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:23.505298 25480 server_base.cc:1034] running on GCE node
I20240502 01:58:23.505599 25480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:23.506103 25480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:23.507252 25480 hybrid_clock.cc:648] HybridClock initialized: now 1714615103507236 us; error 29 us; skew 500 ppm
I20240502 01:58:23.508883 25480 webserver.cc:460] Webserver started at http://127.17.243.8:45521/ using document root <none> and password file <none>
I20240502 01:58:23.509337 25480 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:23.509392 25480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:23.509544 25480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:23.510957 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "bb653087e8554c7591cf547dd969f3be"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.511381 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "bb653087e8554c7591cf547dd969f3be"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.514273 25480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:23.516134 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.516983 25480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:23.517083 25480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bb653087e8554c7591cf547dd969f3be"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:23.517154 25480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:23.532313 25480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:23.532744 25480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:23.532876 25480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:23.533288 25480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:23.534060 25480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:23.534116 25480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.534166 25480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:23.534198 25480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:23.541821 25480 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.8:44673
I20240502 01:58:23.541844 25611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.8:44673 every 8 connection(s)
I20240502 01:58:23.542604 25480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:58:23.544415 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 25480
I20240502 01:58:23.544533 18380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:58:23.550647 25612 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:23.550827 25612 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:23.551261 25612 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:23.552089 24487 ts_manager.cc:194] Registered new tserver with Master: bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:23.552623 24487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.8:54635
I20240502 01:58:23.558266 18380 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:58:23.566681 18380 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:58:23.566741 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 25083
I20240502 01:58:23.573980 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 25216
I20240502 01:58:23.579226 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 25348
I20240502 01:58:23.579782 24811 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
I20240502 01:58:23.583851 18380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5aC8CU/build/debug/bin/kudu with pid 25480
I20240502 01:58:23.748423 24945 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
I20240502 01:58:23.908874 25080 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
I20240502 01:58:27.339388 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:27.358866 24612 tablet_service.cc:1467] Processing CreateTablet for tablet e96b29005a51413082c19a8c79698334 (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.358866 25012 tablet_service.cc:1467] Processing CreateTablet for tablet e96b29005a51413082c19a8c79698334 (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.359819 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96b29005a51413082c19a8c79698334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.359819 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96b29005a51413082c19a8c79698334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.359671 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3c0e77bfb04e71a5dd553245353cfa (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.360450 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3c0e77bfb04e71a5dd553245353cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.362576 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3c0e77bfb04e71a5dd553245353cfa (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.362825 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3c0e77bfb04e71a5dd553245353cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.364630 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 65f16854b1544fb7928482ce0a2f9921 (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.364861 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f16854b1544fb7928482ce0a2f9921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.365355 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 65f16854b1544fb7928482ce0a2f9921 (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.365623 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f16854b1544fb7928482ce0a2f9921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.367228 25633 tablet_bootstrap.cc:492] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:27.369227 25633 tablet_bootstrap.cc:654] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.369271 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3c0e77bfb04e71a5dd553245353cfa (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.370007 25633 log.cc:824] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:27.370085 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3c0e77bfb04e71a5dd553245353cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.370191 24877 tablet_service.cc:1467] Processing CreateTablet for tablet e96b29005a51413082c19a8c79698334 (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.370384 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96b29005a51413082c19a8c79698334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.362196 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 65f16854b1544fb7928482ce0a2f9921 (DEFAULT_TABLE table=rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.371153 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f16854b1544fb7928482ce0a2f9921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.371531 25632 tablet_bootstrap.cc:492] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:27.371589 25633 tablet_bootstrap.cc:492] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:27.371747 25633 ts_tablet_manager.cc:1367] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:27.374687 25635 tablet_bootstrap.cc:492] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:27.374931 25633 raft_consensus.cc:348] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.375140 25633 raft_consensus.cc:374] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.375187 25633 raft_consensus.cc:729] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.375785 25633 consensus_queue.cc:260] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.376715 25633 ts_tablet_manager.cc:1398] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:27.377146 25635 tablet_bootstrap.cc:654] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.377642 25636 tablet_bootstrap.cc:492] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:27.377911 25635 log.cc:824] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:27.378609 25632 tablet_bootstrap.cc:654] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.379169 25632 log.cc:824] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:27.379493 25636 tablet_bootstrap.cc:654] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.379602 25633 tablet_bootstrap.cc:492] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:27.380288 25636 log.cc:824] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:27.380435 25633 tablet_bootstrap.cc:654] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.381023 25632 tablet_bootstrap.cc:492] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:27.381038 25635 tablet_bootstrap.cc:492] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:27.381187 25632 ts_tablet_manager.cc:1367] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:58:27.381187 25635 ts_tablet_manager.cc:1367] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 01:58:27.381487 25633 tablet_bootstrap.cc:492] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:27.381556 25633 ts_tablet_manager.cc:1367] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:27.382081 25633 raft_consensus.cc:348] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.382164 25633 raft_consensus.cc:374] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.382193 25633 raft_consensus.cc:729] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.382294 25633 consensus_queue.cc:260] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.382537 25633 ts_tablet_manager.cc:1398] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.382563 25636 tablet_bootstrap.cc:492] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:27.382673 25633 tablet_bootstrap.cc:492] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:27.382696 25636 ts_tablet_manager.cc:1367] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:27.383405 25633 tablet_bootstrap.cc:654] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.384670 25633 tablet_bootstrap.cc:492] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:27.384743 25633 ts_tablet_manager.cc:1367] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:27.384967 25632 raft_consensus.cc:348] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.384967 25635 raft_consensus.cc:348] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.385156 25632 raft_consensus.cc:374] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.385156 25635 raft_consensus.cc:374] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.385133 25633 raft_consensus.cc:348] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.385206 25632 raft_consensus.cc:729] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.385206 25635 raft_consensus.cc:729] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.385226 25633 raft_consensus.cc:374] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.385253 25633 raft_consensus.cc:729] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.385350 25633 consensus_queue.cc:260] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.385567 25633 ts_tablet_manager.cc:1398] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.385617 25632 consensus_queue.cc:260] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.385617 25635 consensus_queue.cc:260] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.386181 25636 raft_consensus.cc:348] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.386397 25636 raft_consensus.cc:374] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.386471 25635 ts_tablet_manager.cc:1398] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240502 01:58:27.386466 25632 ts_tablet_manager.cc:1398] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:58:27.386492 25636 raft_consensus.cc:729] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.386734 25632 tablet_bootstrap.cc:492] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:27.386881 25636 consensus_queue.cc:260] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.387714 25636 ts_tablet_manager.cc:1398] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:58:27.388084 25635 tablet_bootstrap.cc:492] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:27.389149 25635 tablet_bootstrap.cc:654] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.390353 25635 tablet_bootstrap.cc:492] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:27.390425 25636 tablet_bootstrap.cc:492] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:27.390452 25635 ts_tablet_manager.cc:1367] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:27.390587 25632 tablet_bootstrap.cc:654] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.390995 25635 raft_consensus.cc:348] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.391093 25635 raft_consensus.cc:374] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.391141 25635 raft_consensus.cc:729] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.391278 25635 consensus_queue.cc:260] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.391579 25636 tablet_bootstrap.cc:654] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.391582 25635 ts_tablet_manager.cc:1398] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.392323 25632 tablet_bootstrap.cc:492] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:27.392410 25632 ts_tablet_manager.cc:1367] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:27.392942 25632 raft_consensus.cc:348] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.393088 25632 raft_consensus.cc:374] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.393138 25632 raft_consensus.cc:729] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.393291 25632 consensus_queue.cc:260] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.393575 25632 ts_tablet_manager.cc:1398] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.393116 25636 tablet_bootstrap.cc:492] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:27.393715 25636 ts_tablet_manager.cc:1367] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:27.394651 25636 raft_consensus.cc:348] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.394765 25636 raft_consensus.cc:374] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.394811 25636 raft_consensus.cc:729] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.394925 25636 consensus_queue.cc:260] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.395166 25636 ts_tablet_manager.cc:1398] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:58:27.400743 25081 tablet.cc:2369] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:27.411005 25637 raft_consensus.cc:482] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:27.411136 25637 raft_consensus.cc:504] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.411949 25637 leader_election.cc:288] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:27.416047 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96b29005a51413082c19a8c79698334" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:27.416271 24898 raft_consensus.cc:2454] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:27.416288 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96b29005a51413082c19a8c79698334" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:27.416477 25034 raft_consensus.cc:2454] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:27.416689 24567 leader_election.cc:302] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:27.416908 25637 raft_consensus.cc:2781] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:27.416977 25637 raft_consensus.cc:482] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:27.417011 25637 raft_consensus.cc:3037] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:27.417459 24680 tablet.cc:2369] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:27.417922 25637 raft_consensus.cc:504] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.418262 25637 leader_election.cc:288] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:27.418597 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96b29005a51413082c19a8c79698334" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:27.418718 25034 raft_consensus.cc:3037] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.419731 25034 raft_consensus.cc:2454] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:27.419956 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96b29005a51413082c19a8c79698334" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:27.420020 24564 leader_election.cc:302] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:27.420068 24898 raft_consensus.cc:3037] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.420205 25637 raft_consensus.cc:2781] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:27.420408 25637 raft_consensus.cc:686] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:27.420689 25637 consensus_queue.cc:237] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.421047 24898 raft_consensus.cc:2454] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:27.423339 24487 catalog_manager.cc:5484] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:27.432003 25644 raft_consensus.cc:482] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:27.432125 25644 raft_consensus.cc:504] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.432925 25644 leader_election.cc:288] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:27.436122 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f16854b1544fb7928482ce0a2f9921" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:27.436180 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f16854b1544fb7928482ce0a2f9921" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:27.436219 25034 raft_consensus.cc:2454] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:27.436334 24632 raft_consensus.cc:2454] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:27.436512 24698 leader_election.cc:302] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:27.436712 25644 raft_consensus.cc:2781] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:27.436774 25644 raft_consensus.cc:482] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:27.436820 25644 raft_consensus.cc:3037] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.437561 25644 raft_consensus.cc:504] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.437845 25644 leader_election.cc:288] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:27.438131 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f16854b1544fb7928482ce0a2f9921" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:27.438153 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f16854b1544fb7928482ce0a2f9921" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:27.438228 24632 raft_consensus.cc:3037] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.438225 25034 raft_consensus.cc:3037] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.438838 25034 raft_consensus.cc:2454] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:27.438891 24632 raft_consensus.cc:2454] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:27.439088 24698 leader_election.cc:302] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:27.439249 25644 raft_consensus.cc:2781] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:27.439426 25644 raft_consensus.cc:686] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:27.439675 25644 consensus_queue.cc:237] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.441916 24487 catalog_manager.cc:5484] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:27.489526 24946 tablet.cc:2369] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:27.571287 24812 tablet.cc:2369] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:27.648921 25644 raft_consensus.cc:482] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:27.649063 25644 raft_consensus.cc:504] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.649555 25644 leader_election.cc:288] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:27.650003 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c0e77bfb04e71a5dd553245353cfa" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:27.650169 24632 raft_consensus.cc:2454] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:27.650447 24700 leader_election.cc:302] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:27.650615 25644 raft_consensus.cc:2781] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:27.650694 25644 raft_consensus.cc:482] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:27.650727 25644 raft_consensus.cc:3037] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.651611 25644 raft_consensus.cc:504] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.651921 25644 leader_election.cc:288] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:27.652293 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c0e77bfb04e71a5dd553245353cfa" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:27.652415 24632 raft_consensus.cc:3037] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.653223 24632 raft_consensus.cc:2454] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:27.653216 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c0e77bfb04e71a5dd553245353cfa" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:27.653295 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c0e77bfb04e71a5dd553245353cfa" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:27.653520 24700 leader_election.cc:302] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:27.653682 25644 raft_consensus.cc:2781] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:27.653337 24898 raft_consensus.cc:2454] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:27.653785 25644 raft_consensus.cc:686] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:27.653985 25644 consensus_queue.cc:237] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.655894 24487 catalog_manager.cc:5484] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:27.813319 25658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:27.813513 25658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:27.816483 25658 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:58:27.909688 25034 raft_consensus.cc:1261] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:27.910357 24632 raft_consensus.cc:1261] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:27.910415 25034 raft_consensus.cc:1261] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:27.910804 25651 consensus_queue.cc:1035] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:27.911300 25644 consensus_queue.cc:1035] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:27.911474 24898 raft_consensus.cc:3037] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:27.912082 25655 consensus_queue.cc:1035] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:27.912523 24632 raft_consensus.cc:1261] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:27.912631 24898 raft_consensus.cc:1261] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:27.912904 25644 consensus_queue.cc:1035] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:27.916723 25655 consensus_queue.cc:1035] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:27.919148 25689 mvcc.cc:205] Tried to move back new op lower bound from 7023063481995063296 to 7023063480952160256. Current Snapshot: MvccSnapshot[applied={T|T < 7023063481995063296}]
I20240502 01:58:27.919396 25692 mvcc.cc:205] Tried to move back new op lower bound from 7023063481995063296 to 7023063480952160256. Current Snapshot: MvccSnapshot[applied={T|T < 7023063481995063296}]
I20240502 01:58:27.920534 24898 raft_consensus.cc:1261] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:27.920960 25651 consensus_queue.cc:1035] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:27.934195 25694 mvcc.cc:205] Tried to move back new op lower bound from 7023063481995300864 to 7023063479997710336. Current Snapshot: MvccSnapshot[applied={T|T < 7023063481995300864}]
I20240502 01:58:27.942723 25705 mvcc.cc:205] Tried to move back new op lower bound from 7023063482019479552 to 7023063480952160256. Current Snapshot: MvccSnapshot[applied={T|T < 7023063481995063296}]
INSERT report
rows total: 2000
time total: 118.457 ms
time per row: 0.0592283 ms
I20240502 01:58:27.972137 24484 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:27.977388 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 48085cebf94e4f549e6174a88acdcc59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.977689 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48085cebf94e4f549e6174a88acdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.977645 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 48085cebf94e4f549e6174a88acdcc59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.977767 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 26422e12ddf64e27b3913c60800e5a56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.977893 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48085cebf94e4f549e6174a88acdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.977916 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 48085cebf94e4f549e6174a88acdcc59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.978211 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48085cebf94e4f549e6174a88acdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.978480 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26422e12ddf64e27b3913c60800e5a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.980118 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 44be529dec974eac8419efe9c19b30bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.980343 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44be529dec974eac8419efe9c19b30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.980332 25724 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:27.980389 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 26422e12ddf64e27b3913c60800e5a56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.980585 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26422e12ddf64e27b3913c60800e5a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.981102 25724 tablet_bootstrap.cc:654] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.982120 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 44be529dec974eac8419efe9c19b30bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.982323 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44be529dec974eac8419efe9c19b30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.982364 25724 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:27.982439 25724 ts_tablet_manager.cc:1367] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:27.982478 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 44be529dec974eac8419efe9c19b30bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.982688 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44be529dec974eac8419efe9c19b30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.982910 25724 raft_consensus.cc:348] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.983003 25724 raft_consensus.cc:374] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.983031 25724 raft_consensus.cc:729] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.983150 25724 consensus_queue.cc:260] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.983628 25724 ts_tablet_manager.cc:1398] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.983839 25727 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:27.984570 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 26422e12ddf64e27b3913c60800e5a56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:27.984954 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26422e12ddf64e27b3913c60800e5a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:27.985002 25725 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:27.985770 25725 tablet_bootstrap.cc:654] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.985770 25727 tablet_bootstrap.cc:654] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.986685 25724 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:27.987182 25725 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:27.987260 25725 ts_tablet_manager.cc:1367] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:27.987514 25724 tablet_bootstrap.cc:654] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.987614 25725 raft_consensus.cc:348] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.987702 25725 raft_consensus.cc:374] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.987756 25725 raft_consensus.cc:729] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.987934 25725 consensus_queue.cc:260] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.988159 25723 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:27.988235 25725 ts_tablet_manager.cc:1398] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.988444 25725 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:27.988823 25727 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:27.988893 25723 tablet_bootstrap.cc:654] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.988906 25727 ts_tablet_manager.cc:1367] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:58:27.989274 25725 tablet_bootstrap.cc:654] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.989295 25727 raft_consensus.cc:348] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.989380 25727 raft_consensus.cc:374] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.989411 25727 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.989580 25724 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:27.989537 25727 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.989636 25724 ts_tablet_manager.cc:1367] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:27.989791 25727 ts_tablet_manager.cc:1398] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:27.990015 25724 raft_consensus.cc:348] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.990085 25724 raft_consensus.cc:374] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.990105 25724 raft_consensus.cc:729] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.990192 25724 consensus_queue.cc:260] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.990428 25724 ts_tablet_manager.cc:1398] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.990589 25724 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:27.991386 25724 tablet_bootstrap.cc:654] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.992434 25723 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:27.992545 25723 ts_tablet_manager.cc:1367] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:27.992630 25725 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:27.992717 25725 ts_tablet_manager.cc:1367] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:27.993005 25723 raft_consensus.cc:348] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.993121 25723 raft_consensus.cc:374] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.993173 25723 raft_consensus.cc:729] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.993149 25725 raft_consensus.cc:348] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.993255 25725 raft_consensus.cc:374] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.993330 25725 raft_consensus.cc:729] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.993360 25723 consensus_queue.cc:260] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.993479 25725 consensus_queue.cc:260] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.993631 25723 ts_tablet_manager.cc:1398] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.993780 25725 ts_tablet_manager.cc:1398] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.993808 25723 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:27.993996 25725 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:27.994546 25723 tablet_bootstrap.cc:654] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.994863 25725 tablet_bootstrap.cc:654] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:27.995378 25723 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:27.995453 25723 ts_tablet_manager.cc:1367] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:27.995846 25723 raft_consensus.cc:348] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.995931 25723 raft_consensus.cc:374] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.995960 25723 raft_consensus.cc:729] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.996069 25723 consensus_queue.cc:260] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:27.996218 25724 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:27.996280 25724 ts_tablet_manager.cc:1367] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:27.996305 25723 ts_tablet_manager.cc:1398] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.996646 25724 raft_consensus.cc:348] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.996743 25724 raft_consensus.cc:374] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.996776 25724 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.996894 25724 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.997093 25724 ts_tablet_manager.cc:1398] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.997741 25725 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:27.997825 25725 ts_tablet_manager.cc:1367] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:27.998337 25725 raft_consensus.cc:348] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.998804 25725 raft_consensus.cc:374] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:27.998863 25725 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:27.998991 25725 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:27.999380 25725 ts_tablet_manager.cc:1398] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:28.115911 25707 raft_consensus.cc:482] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:28.116052 25707 raft_consensus.cc:504] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.116475 25707 leader_election.cc:288] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:28.116864 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26422e12ddf64e27b3913c60800e5a56" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:28.116880 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26422e12ddf64e27b3913c60800e5a56" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:28.117012 24898 raft_consensus.cc:2454] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:28.117018 25032 raft_consensus.cc:2454] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:28.117295 24698 leader_election.cc:302] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.117455 25707 raft_consensus.cc:2781] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:28.117527 25707 raft_consensus.cc:482] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:28.117558 25707 raft_consensus.cc:3037] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.118409 25707 raft_consensus.cc:504] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.118727 25707 leader_election.cc:288] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:28.119032 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26422e12ddf64e27b3913c60800e5a56" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:28.119092 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26422e12ddf64e27b3913c60800e5a56" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:28.119150 24898 raft_consensus.cc:3037] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.119195 25032 raft_consensus.cc:3037] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.120045 24898 raft_consensus.cc:2454] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:28.120043 25032 raft_consensus.cc:2454] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:28.120330 24701 leader_election.cc:302] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:28.120486 25707 raft_consensus.cc:2781] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:28.120604 25707 raft_consensus.cc:686] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:28.120743 25707 consensus_queue.cc:237] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.122284 24479 catalog_manager.cc:5484] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:28.130546 25704 raft_consensus.cc:482] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:28.130633 25704 raft_consensus.cc:504] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.131563 25704 leader_election.cc:288] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:28.135353 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:28.135526 24765 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:28.135648 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:28.135789 24631 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:28.135900 24833 leader_election.cc:302] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:28.136139 25704 raft_consensus.cc:2781] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:28.136196 25704 raft_consensus.cc:482] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:28.136241 25704 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.136952 25704 raft_consensus.cc:504] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.137238 25704 leader_election.cc:288] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:28.137542 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:28.137652 24631 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.137655 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:28.137746 24765 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.138376 24631 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:28.138504 24765 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:28.138653 24832 leader_election.cc:302] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:28.138826 25704 raft_consensus.cc:2781] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:28.138917 25704 raft_consensus.cc:686] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:28.139176 25704 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.141332 24479 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:28.214007 25704 raft_consensus.cc:482] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:28.214131 25704 raft_consensus.cc:504] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.214589 25704 leader_election.cc:288] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:28.215080 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48085cebf94e4f549e6174a88acdcc59" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:28.215236 24765 raft_consensus.cc:2454] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:28.215523 24833 leader_election.cc:302] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:28.215704 25704 raft_consensus.cc:2781] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:28.215750 25704 raft_consensus.cc:482] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:28.215785 25704 raft_consensus.cc:3037] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.216738 25704 raft_consensus.cc:504] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.217088 25704 leader_election.cc:288] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:28.217406 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48085cebf94e4f549e6174a88acdcc59" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:28.217511 24765 raft_consensus.cc:3037] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.218449 24765 raft_consensus.cc:2454] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:28.218592 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48085cebf94e4f549e6174a88acdcc59" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:28.218628 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48085cebf94e4f549e6174a88acdcc59" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:28.218719 25032 raft_consensus.cc:2454] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:28.219851 24833 leader_election.cc:302] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters: e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:58:28.220033 25704 raft_consensus.cc:2781] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:28.220120 25704 raft_consensus.cc:686] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:28.220255 25704 consensus_queue.cc:237] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.221693 24479 catalog_manager.cc:5484] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:28.373135 25742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:28.373342 25742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:28.376271 25742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:58:28.450364 24765 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.450917 24631 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.450889 25704 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:28.451455 25704 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:28.456049 25032 raft_consensus.cc:3037] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.456099 25034 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.456497 25707 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:28.457108 25032 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.458340 24898 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.458637 24764 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.458680 25710 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:28.458884 25707 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:28.459928 25710 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.6671 ms
time per row: 0.0498336 ms
I20240502 01:58:28.514253 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:28.519289 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 00616380fd3941159bfd4c2a6bba5761 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.519363 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 00616380fd3941159bfd4c2a6bba5761 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.519593 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00616380fd3941159bfd4c2a6bba5761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.519654 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00616380fd3941159bfd4c2a6bba5761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.521203 25798 tablet_bootstrap.cc:492] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:28.521250 25799 tablet_bootstrap.cc:492] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:28.521373 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 00616380fd3941159bfd4c2a6bba5761 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.521621 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00616380fd3941159bfd4c2a6bba5761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.521700 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 7a51df555e1143b8b8db1d70f7407a1f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.521898 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a51df555e1143b8b8db1d70f7407a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.522075 25798 tablet_bootstrap.cc:654] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.522076 25799 tablet_bootstrap.cc:654] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.522207 24875 tablet_service.cc:1467] Processing CreateTablet for tablet fc67bd4c105d49c3afe3940637db5ab9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.522441 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc67bd4c105d49c3afe3940637db5ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.523039 25799 tablet_bootstrap.cc:492] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:28.523108 25799 ts_tablet_manager.cc:1367] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:28.523506 25799 raft_consensus.cc:348] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.523600 25799 raft_consensus.cc:374] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.523631 25799 raft_consensus.cc:729] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.523743 25799 consensus_queue.cc:260] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.523856 25798 tablet_bootstrap.cc:492] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:28.523927 25798 ts_tablet_manager.cc:1367] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:28.524250 25798 raft_consensus.cc:348] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.524338 25798 raft_consensus.cc:374] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.524369 25798 raft_consensus.cc:729] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.524369 25799 ts_tablet_manager.cc:1398] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:28.524516 25799 tablet_bootstrap.cc:492] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:28.524477 25798 consensus_queue.cc:260] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.524718 25798 ts_tablet_manager.cc:1398] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:28.524869 24612 tablet_service.cc:1467] Processing CreateTablet for tablet fc67bd4c105d49c3afe3940637db5ab9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.525075 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc67bd4c105d49c3afe3940637db5ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.525276 25799 tablet_bootstrap.cc:654] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.526204 25799 tablet_bootstrap.cc:492] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:28.526225 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 7a51df555e1143b8b8db1d70f7407a1f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.526279 25799 ts_tablet_manager.cc:1367] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:28.526389 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a51df555e1143b8b8db1d70f7407a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.526671 25799 raft_consensus.cc:348] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.526758 25799 raft_consensus.cc:374] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.526778 25799 raft_consensus.cc:729] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.526867 25799 consensus_queue.cc:260] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.527076 25799 ts_tablet_manager.cc:1398] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:28.527992 25804 tablet_bootstrap.cc:492] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:28.528738 25804 tablet_bootstrap.cc:654] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.529546 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 7a51df555e1143b8b8db1d70f7407a1f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.529783 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a51df555e1143b8b8db1d70f7407a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.531591 25798 tablet_bootstrap.cc:492] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:28.531757 25012 tablet_service.cc:1467] Processing CreateTablet for tablet fc67bd4c105d49c3afe3940637db5ab9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:28.531956 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc67bd4c105d49c3afe3940637db5ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:28.532287 25798 tablet_bootstrap.cc:654] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.533130 25798 tablet_bootstrap.cc:492] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:28.533203 25798 ts_tablet_manager.cc:1367] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:28.533201 25804 tablet_bootstrap.cc:492] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:28.533279 25804 ts_tablet_manager.cc:1367] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:28.533639 25802 tablet_bootstrap.cc:492] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:28.533617 25798 raft_consensus.cc:348] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.533704 25798 raft_consensus.cc:374] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.533740 25798 raft_consensus.cc:729] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.533707 25804 raft_consensus.cc:348] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.533788 25804 raft_consensus.cc:374] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.533861 25798 consensus_queue.cc:260] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.533891 25804 raft_consensus.cc:729] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.534121 25798 ts_tablet_manager.cc:1398] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:28.534163 25804 consensus_queue.cc:260] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.534266 25798 tablet_bootstrap.cc:492] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:28.534410 25804 ts_tablet_manager.cc:1398] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:28.534579 25802 tablet_bootstrap.cc:654] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.534576 25804 tablet_bootstrap.cc:492] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:28.535061 25798 tablet_bootstrap.cc:654] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.535341 25804 tablet_bootstrap.cc:654] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.535522 25802 tablet_bootstrap.cc:492] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:28.535593 25802 ts_tablet_manager.cc:1367] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:28.535971 25802 raft_consensus.cc:348] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.536064 25802 raft_consensus.cc:374] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.536093 25802 raft_consensus.cc:729] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.536196 25802 consensus_queue.cc:260] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.536437 25802 ts_tablet_manager.cc:1398] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:28.537201 25798 tablet_bootstrap.cc:492] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:28.537277 25798 ts_tablet_manager.cc:1367] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:28.537652 25798 raft_consensus.cc:348] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.537715 25798 raft_consensus.cc:374] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.537734 25798 raft_consensus.cc:729] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.537829 25798 consensus_queue.cc:260] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.538074 25798 ts_tablet_manager.cc:1398] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:28.538928 25804 tablet_bootstrap.cc:492] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:28.539031 25804 ts_tablet_manager.cc:1367] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:28.540077 25804 raft_consensus.cc:348] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.540158 25804 raft_consensus.cc:374] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.540176 25804 raft_consensus.cc:729] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.540274 25804 consensus_queue.cc:260] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.540493 25804 ts_tablet_manager.cc:1398] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:28.540623 25804 tablet_bootstrap.cc:492] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:28.541328 25804 tablet_bootstrap.cc:654] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:28.542232 25804 tablet_bootstrap.cc:492] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:28.542304 25804 ts_tablet_manager.cc:1367] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:28.542650 25804 raft_consensus.cc:348] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.542733 25804 raft_consensus.cc:374] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:28.542762 25804 raft_consensus.cc:729] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:28.542898 25804 consensus_queue.cc:260] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.543149 25804 ts_tablet_manager.cc:1398] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:28.580256 25651 raft_consensus.cc:482] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:28.580363 25651 raft_consensus.cc:504] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.580832 25651 leader_election.cc:288] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:28.581373 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a51df555e1143b8b8db1d70f7407a1f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:28.581521 25034 raft_consensus.cc:2454] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:28.581830 24564 leader_election.cc:302] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.582027 25651 raft_consensus.cc:2781] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:28.582072 25651 raft_consensus.cc:482] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:28.582108 25651 raft_consensus.cc:3037] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.583096 25651 raft_consensus.cc:504] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.583428 25651 leader_election.cc:288] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:28.583776 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a51df555e1143b8b8db1d70f7407a1f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:28.583871 25034 raft_consensus.cc:3037] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.584729 25034 raft_consensus.cc:2454] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:28.585011 24564 leader_election.cc:302] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.585163 25651 raft_consensus.cc:2781] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:28.585179 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a51df555e1143b8b8db1d70f7407a1f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:28.585227 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a51df555e1143b8b8db1d70f7407a1f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:28.585320 25651 raft_consensus.cc:686] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:28.585328 24764 raft_consensus.cc:2454] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:28.585454 25651 consensus_queue.cc:237] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.586889 24487 catalog_manager.cc:5484] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:28.689536 25697 raft_consensus.cc:482] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:28.689661 25697 raft_consensus.cc:504] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.690768 25697 leader_election.cc:288] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:28.694329 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00616380fd3941159bfd4c2a6bba5761" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:28.694399 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00616380fd3941159bfd4c2a6bba5761" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:28.694463 24764 raft_consensus.cc:2454] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:28.694551 24631 raft_consensus.cc:2454] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:28.694809 24966 leader_election.cc:302] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.695048 25697 raft_consensus.cc:2781] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:28.695108 25697 raft_consensus.cc:482] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:28.695142 25697 raft_consensus.cc:3037] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.696146 25697 raft_consensus.cc:504] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.696422 25697 leader_election.cc:288] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:28.696732 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00616380fd3941159bfd4c2a6bba5761" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:28.696761 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00616380fd3941159bfd4c2a6bba5761" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:28.696854 24631 raft_consensus.cc:3037] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.696858 24764 raft_consensus.cc:3037] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.697522 24764 raft_consensus.cc:2454] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:28.697621 24631 raft_consensus.cc:2454] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:28.697806 24966 leader_election.cc:302] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.698024 25697 raft_consensus.cc:2781] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:28.698184 25697 raft_consensus.cc:686] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:28.698436 25697 consensus_queue.cc:237] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:28.700681 24487 catalog_manager.cc:5484] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:28.770936 25697 raft_consensus.cc:482] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:28.771037 25697 raft_consensus.cc:504] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.771438 25697 leader_election.cc:288] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:28.771829 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc67bd4c105d49c3afe3940637db5ab9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:28.771986 24631 raft_consensus.cc:2454] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:28.772284 24965 leader_election.cc:302] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.772464 25697 raft_consensus.cc:2781] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:28.772532 25697 raft_consensus.cc:482] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:28.772580 25697 raft_consensus.cc:3037] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.773418 25697 raft_consensus.cc:504] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.773770 25697 leader_election.cc:288] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:28.774156 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc67bd4c105d49c3afe3940637db5ab9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:28.774281 24631 raft_consensus.cc:3037] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.775130 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc67bd4c105d49c3afe3940637db5ab9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:28.775187 24631 raft_consensus.cc:2454] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:28.775231 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc67bd4c105d49c3afe3940637db5ab9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:28.775254 24898 raft_consensus.cc:2454] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:28.775449 24965 leader_election.cc:302] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:28.775647 25697 raft_consensus.cc:2781] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:28.775738 25697 raft_consensus.cc:686] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:28.775862 25697 consensus_queue.cc:237] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:28.777364 24487 catalog_manager.cc:5484] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:28.923241 25820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:28.923439 25820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:28.926307 25820 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:58:28.996572 24764 raft_consensus.cc:3037] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:28.996694 25034 raft_consensus.cc:1261] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.997103 25812 consensus_queue.cc:1035] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:28.997659 24764 raft_consensus.cc:1261] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.998199 24764 raft_consensus.cc:1261] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:28.998628 25697 consensus_queue.cc:1035] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.000397 25651 consensus_queue.cc:1035] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.001276 24631 raft_consensus.cc:1261] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.002368 24898 raft_consensus.cc:3037] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.002415 24631 raft_consensus.cc:1261] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.002487 25697 consensus_queue.cc:1035] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.002919 25697 consensus_queue.cc:1035] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.003414 24898 raft_consensus.cc:1261] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.003787 25697 consensus_queue.cc:1035] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 96.7405 ms
time per row: 0.0483702 ms
I20240502 01:58:29.062070 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:29.066984 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 49d6b2dc45b84ed99ef017b736db25a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.067265 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d6b2dc45b84ed99ef017b736db25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.067201 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 49d6b2dc45b84ed99ef017b736db25a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.067467 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d6b2dc45b84ed99ef017b736db25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.068881 25875 tablet_bootstrap.cc:492] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:29.069450 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 488816b0d5d542bcb5fa12ff7e70fd40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.069478 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 488816b0d5d542bcb5fa12ff7e70fd40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.069679 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488816b0d5d542bcb5fa12ff7e70fd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.069707 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488816b0d5d542bcb5fa12ff7e70fd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.070225 25876 tablet_bootstrap.cc:492] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:29.070210 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 49d6b2dc45b84ed99ef017b736db25a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.070210 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 488816b0d5d542bcb5fa12ff7e70fd40 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.070446 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d6b2dc45b84ed99ef017b736db25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.070446 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488816b0d5d542bcb5fa12ff7e70fd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.071394 25876 tablet_bootstrap.cc:654] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.072229 24610 tablet_service.cc:1467] Processing CreateTablet for tablet b3edec06b3d8450a8cc3002be70c848a (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.072372 25876 tablet_bootstrap.cc:492] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:29.072443 25876 ts_tablet_manager.cc:1367] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:29.072448 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3edec06b3d8450a8cc3002be70c848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.072763 25879 tablet_bootstrap.cc:492] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:29.072844 25876 raft_consensus.cc:348] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.072933 25876 raft_consensus.cc:374] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.072957 25876 raft_consensus.cc:729] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.073076 25876 consensus_queue.cc:260] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.073336 25876 ts_tablet_manager.cc:1398] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.073488 25876 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:29.073737 25879 tablet_bootstrap.cc:654] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.074270 25876 tablet_bootstrap.cc:654] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.069869 25012 tablet_service.cc:1467] Processing CreateTablet for tablet b3edec06b3d8450a8cc3002be70c848a (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.074445 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3edec06b3d8450a8cc3002be70c848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.075763 25879 tablet_bootstrap.cc:492] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:29.075846 25879 ts_tablet_manager.cc:1367] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:29.076012 24875 tablet_service.cc:1467] Processing CreateTablet for tablet b3edec06b3d8450a8cc3002be70c848a (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.076207 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3edec06b3d8450a8cc3002be70c848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.076260 25879 raft_consensus.cc:348] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.076344 25879 raft_consensus.cc:374] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.076371 25879 raft_consensus.cc:729] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.076433 25875 tablet_bootstrap.cc:654] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.076493 25879 consensus_queue.cc:260] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.076738 25879 ts_tablet_manager.cc:1398] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.076886 25879 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:29.077450 25875 tablet_bootstrap.cc:492] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:29.077584 25875 ts_tablet_manager.cc:1367] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:29.077657 25878 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:29.077667 25879 tablet_bootstrap.cc:654] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.078052 25875 raft_consensus.cc:348] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.078210 25875 raft_consensus.cc:374] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.078305 25875 raft_consensus.cc:729] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.078497 25878 tablet_bootstrap.cc:654] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.078469 25875 consensus_queue.cc:260] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.078824 25875 ts_tablet_manager.cc:1398] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.079056 25875 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:29.079423 25876 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:29.079515 25876 ts_tablet_manager.cc:1367] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:29.079829 25878 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:29.079937 25875 tablet_bootstrap.cc:654] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.079914 25876 raft_consensus.cc:348] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.079967 25878 ts_tablet_manager.cc:1367] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:29.080004 25876 raft_consensus.cc:374] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.080034 25876 raft_consensus.cc:729] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.080142 25876 consensus_queue.cc:260] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.080406 25876 ts_tablet_manager.cc:1398] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.080485 25878 raft_consensus.cc:348] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.080597 25878 raft_consensus.cc:374] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.080674 25878 raft_consensus.cc:729] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.081123 25879 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:29.081203 25879 ts_tablet_manager.cc:1367] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:29.081380 25878 consensus_queue.cc:260] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.081629 25879 raft_consensus.cc:348] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.081696 25878 ts_tablet_manager.cc:1398] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:29.081727 25879 raft_consensus.cc:374] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.081760 25879 raft_consensus.cc:729] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.081853 25878 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:29.081861 25879 consensus_queue.cc:260] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.082149 25879 ts_tablet_manager.cc:1398] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:29.082700 25878 tablet_bootstrap.cc:654] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.083415 25875 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:29.083540 25875 ts_tablet_manager.cc:1367] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:29.083941 25875 raft_consensus.cc:348] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.084074 25878 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:29.084095 25875 raft_consensus.cc:374] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.084131 25875 raft_consensus.cc:729] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.084158 25878 ts_tablet_manager.cc:1367] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:29.084261 25875 consensus_queue.cc:260] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.084556 25878 raft_consensus.cc:348] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.084653 25878 raft_consensus.cc:374] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.084687 25878 raft_consensus.cc:729] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.084796 25878 consensus_queue.cc:260] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.084887 25875 ts_tablet_manager.cc:1398] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.085034 25875 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:29.085034 25878 ts_tablet_manager.cc:1398] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.085776 25875 tablet_bootstrap.cc:654] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.087229 25875 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:29.087313 25875 ts_tablet_manager.cc:1367] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:29.087668 25875 raft_consensus.cc:348] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.087779 25875 raft_consensus.cc:374] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.087812 25875 raft_consensus.cc:729] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.087919 25875 consensus_queue.cc:260] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.088130 25875 ts_tablet_manager.cc:1398] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.233320 25817 raft_consensus.cc:482] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:29.233444 25817 raft_consensus.cc:504] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.233853 25817 leader_election.cc:288] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.234243 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488816b0d5d542bcb5fa12ff7e70fd40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:29.234304 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488816b0d5d542bcb5fa12ff7e70fd40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:29.234373 24898 raft_consensus.cc:2454] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:29.234450 24765 raft_consensus.cc:2454] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:29.234697 24966 leader_election.cc:302] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.234874 25817 raft_consensus.cc:2781] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:29.234927 25817 raft_consensus.cc:482] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:29.234977 25817 raft_consensus.cc:3037] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.235770 25817 raft_consensus.cc:504] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.236064 25817 leader_election.cc:288] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.236399 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488816b0d5d542bcb5fa12ff7e70fd40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:29.236420 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488816b0d5d542bcb5fa12ff7e70fd40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:29.236512 24898 raft_consensus.cc:3037] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.236524 24765 raft_consensus.cc:3037] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.237195 24765 raft_consensus.cc:2454] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:29.237251 24898 raft_consensus.cc:2454] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:29.237455 24966 leader_election.cc:302] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:29.237603 25817 raft_consensus.cc:2781] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:29.237694 25817 raft_consensus.cc:686] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:29.237821 25817 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:29.239382 24487 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:29.314930 25790 raft_consensus.cc:482] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:29.315050 25790 raft_consensus.cc:504] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.315397 25790 leader_election.cc:288] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:29.315858 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3edec06b3d8450a8cc3002be70c848a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:29.315858 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3edec06b3d8450a8cc3002be70c848a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:29.316008 25034 raft_consensus.cc:2454] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:29.316013 24632 raft_consensus.cc:2454] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:29.316283 24832 leader_election.cc:302] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.316498 25790 raft_consensus.cc:2781] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:29.316571 25790 raft_consensus.cc:482] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:29.316622 25790 raft_consensus.cc:3037] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.317518 25790 raft_consensus.cc:504] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.317859 25790 leader_election.cc:288] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:29.318199 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3edec06b3d8450a8cc3002be70c848a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:29.318248 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3edec06b3d8450a8cc3002be70c848a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:29.318298 24632 raft_consensus.cc:3037] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.318368 25034 raft_consensus.cc:3037] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.319022 24632 raft_consensus.cc:2454] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:29.319092 25034 raft_consensus.cc:2454] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:29.319284 24832 leader_election.cc:302] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.319531 25790 raft_consensus.cc:2781] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:29.319679 25790 raft_consensus.cc:686] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:29.319808 25790 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:29.321305 24487 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:29.350569 25651 raft_consensus.cc:482] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:29.350672 25651 raft_consensus.cc:504] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.351006 25651 leader_election.cc:288] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.351454 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6b2dc45b84ed99ef017b736db25a5" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:29.351516 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6b2dc45b84ed99ef017b736db25a5" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:29.351583 24898 raft_consensus.cc:2454] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:29.351651 24765 raft_consensus.cc:2454] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:29.351874 24567 leader_election.cc:302] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.352059 25651 raft_consensus.cc:2781] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:29.352115 25651 raft_consensus.cc:482] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:29.352156 25651 raft_consensus.cc:3037] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.352871 25651 raft_consensus.cc:504] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.353159 25651 leader_election.cc:288] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.353454 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6b2dc45b84ed99ef017b736db25a5" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:29.353559 24898 raft_consensus.cc:3037] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.353632 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6b2dc45b84ed99ef017b736db25a5" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:29.353741 24765 raft_consensus.cc:3037] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.354503 24898 raft_consensus.cc:2454] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:29.354630 24765 raft_consensus.cc:2454] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:29.354804 24567 leader_election.cc:302] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.354962 25651 raft_consensus.cc:2781] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:29.355041 25651 raft_consensus.cc:686] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:29.355171 25651 consensus_queue.cc:237] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.356809 24487 catalog_manager.cc:5484] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:29.463768 25893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:29.463970 25893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:29.466944 25893 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:58:29.548101 24765 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.548478 24898 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.548533 25817 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.549082 25817 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.553561 24764 raft_consensus.cc:1261] T 49d6b2dc45b84ed99ef017b736db25a5 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.553874 24897 raft_consensus.cc:1261] T 49d6b2dc45b84ed99ef017b736db25a5 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.554389 25812 consensus_queue.cc:1035] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.554541 25651 consensus_queue.cc:1035] T 49d6b2dc45b84ed99ef017b736db25a5 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.556926 25034 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.557790 25790 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:29.558825 24632 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:29.559234 25790 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.714 ms
time per row: 0.0598569 ms
I20240502 01:58:29.625056 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:29.630283 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 4844b449d79d425f86b3076f9f43404f (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.630407 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 4844b449d79d425f86b3076f9f43404f (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.630576 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4844b449d79d425f86b3076f9f43404f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.630678 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4844b449d79d425f86b3076f9f43404f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.631148 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 4844b449d79d425f86b3076f9f43404f (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.631398 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4844b449d79d425f86b3076f9f43404f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.631445 24612 tablet_service.cc:1467] Processing CreateTablet for tablet b5340b3277654975bbbdfb4312b7c00d (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.631680 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5340b3277654975bbbdfb4312b7c00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.633070 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 803c34f634d6480a862eeb642aaee3cc (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.633289 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803c34f634d6480a862eeb642aaee3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.633872 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 803c34f634d6480a862eeb642aaee3cc (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.633983 25012 tablet_service.cc:1467] Processing CreateTablet for tablet b5340b3277654975bbbdfb4312b7c00d (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.634148 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803c34f634d6480a862eeb642aaee3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.634166 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 803c34f634d6480a862eeb642aaee3cc (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.634356 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5340b3277654975bbbdfb4312b7c00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.634361 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803c34f634d6480a862eeb642aaee3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.635192 24745 tablet_service.cc:1467] Processing CreateTablet for tablet b5340b3277654975bbbdfb4312b7c00d (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:29.635407 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5340b3277654975bbbdfb4312b7c00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:29.636232 25955 tablet_bootstrap.cc:492] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:29.636340 25958 tablet_bootstrap.cc:492] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:29.636925 25955 tablet_bootstrap.cc:654] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.637146 25958 tablet_bootstrap.cc:654] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.637588 25956 tablet_bootstrap.cc:492] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:29.637624 25957 tablet_bootstrap.cc:492] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:29.638410 25957 tablet_bootstrap.cc:654] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.638430 25958 tablet_bootstrap.cc:492] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:29.638445 25956 tablet_bootstrap.cc:654] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.638561 25958 ts_tablet_manager.cc:1367] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:29.639014 25958 raft_consensus.cc:348] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.639153 25958 raft_consensus.cc:374] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.639204 25958 raft_consensus.cc:729] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.639350 25958 consensus_queue.cc:260] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.639380 25955 tablet_bootstrap.cc:492] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:29.639380 25956 tablet_bootstrap.cc:492] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:29.639459 25955 ts_tablet_manager.cc:1367] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:29.639459 25956 ts_tablet_manager.cc:1367] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:29.639614 25958 ts_tablet_manager.cc:1398] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.639804 25958 tablet_bootstrap.cc:492] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:29.639856 25955 raft_consensus.cc:348] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.639969 25955 raft_consensus.cc:374] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.640002 25955 raft_consensus.cc:729] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.640105 25955 consensus_queue.cc:260] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.640281 25956 raft_consensus.cc:348] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.640348 25955 ts_tablet_manager.cc:1398] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.640367 25956 raft_consensus.cc:374] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.640406 25956 raft_consensus.cc:729] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.640501 25955 tablet_bootstrap.cc:492] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:29.640606 25958 tablet_bootstrap.cc:654] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.640515 25956 consensus_queue.cc:260] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.641134 25956 ts_tablet_manager.cc:1398] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:29.641294 25955 tablet_bootstrap.cc:654] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.641325 25956 tablet_bootstrap.cc:492] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:29.641578 25958 tablet_bootstrap.cc:492] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:29.641649 25958 ts_tablet_manager.cc:1367] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:29.642148 25956 tablet_bootstrap.cc:654] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.642095 25958 raft_consensus.cc:348] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.642191 25958 raft_consensus.cc:374] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.642221 25958 raft_consensus.cc:729] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.642323 25958 consensus_queue.cc:260] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.642571 25958 ts_tablet_manager.cc:1398] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.643251 25956 tablet_bootstrap.cc:492] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:29.643363 25956 ts_tablet_manager.cc:1367] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:29.643573 25957 tablet_bootstrap.cc:492] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:29.643658 25957 ts_tablet_manager.cc:1367] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:29.643771 25956 raft_consensus.cc:348] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.643859 25956 raft_consensus.cc:374] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.643887 25956 raft_consensus.cc:729] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.643998 25956 consensus_queue.cc:260] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.644076 25957 raft_consensus.cc:348] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.644177 25957 raft_consensus.cc:374] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.644207 25957 raft_consensus.cc:729] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.644255 25956 ts_tablet_manager.cc:1398] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.644315 25957 consensus_queue.cc:260] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.644582 25957 ts_tablet_manager.cc:1398] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.644789 25957 tablet_bootstrap.cc:492] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:29.645352 25955 tablet_bootstrap.cc:492] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:29.645422 25955 ts_tablet_manager.cc:1367] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:29.645588 25957 tablet_bootstrap.cc:654] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.645810 25955 raft_consensus.cc:348] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.645913 25955 raft_consensus.cc:374] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.645963 25955 raft_consensus.cc:729] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.646070 25955 consensus_queue.cc:260] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.646322 25955 ts_tablet_manager.cc:1398] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.646468 25955 tablet_bootstrap.cc:492] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:29.647471 25955 tablet_bootstrap.cc:654] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:29.648451 25955 tablet_bootstrap.cc:492] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:29.648569 25955 ts_tablet_manager.cc:1367] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:29.648828 25957 tablet_bootstrap.cc:492] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:29.648924 25957 ts_tablet_manager.cc:1367] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:29.648967 25955 raft_consensus.cc:348] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.649114 25955 raft_consensus.cc:374] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.649161 25955 raft_consensus.cc:729] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.649253 25955 consensus_queue.cc:260] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.649317 25957 raft_consensus.cc:348] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.649513 25955 ts_tablet_manager.cc:1398] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:29.649510 25957 raft_consensus.cc:374] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:29.649551 25957 raft_consensus.cc:729] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:29.649709 25957 consensus_queue.cc:260] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.650072 25957 ts_tablet_manager.cc:1398] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:29.742367 25890 raft_consensus.cc:482] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:29.742486 25890 raft_consensus.cc:504] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.742880 25890 leader_election.cc:288] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.743255 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4844b449d79d425f86b3076f9f43404f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:29.743257 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4844b449d79d425f86b3076f9f43404f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:29.743400 24632 raft_consensus.cc:2454] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:29.743400 24765 raft_consensus.cc:2454] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:29.743705 24833 leader_election.cc:302] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.743901 25890 raft_consensus.cc:2781] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:29.743978 25890 raft_consensus.cc:482] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:29.744017 25890 raft_consensus.cc:3037] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.745437 25890 raft_consensus.cc:504] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.745807 25890 leader_election.cc:288] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.746152 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4844b449d79d425f86b3076f9f43404f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:29.746212 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4844b449d79d425f86b3076f9f43404f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:29.746260 24632 raft_consensus.cc:3037] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.746330 24765 raft_consensus.cc:3037] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.747035 24632 raft_consensus.cc:2454] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:29.747258 24765 raft_consensus.cc:2454] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:29.747308 24832 leader_election.cc:302] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.747475 25890 raft_consensus.cc:2781] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:29.747567 25890 raft_consensus.cc:686] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:29.747709 25890 consensus_queue.cc:237] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.749534 24482 catalog_manager.cc:5484] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:29.806749 25651 raft_consensus.cc:482] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:29.806886 25651 raft_consensus.cc:504] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.807325 25651 leader_election.cc:288] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.807729 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5340b3277654975bbbdfb4312b7c00d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:29.807768 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5340b3277654975bbbdfb4312b7c00d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:29.807884 25034 raft_consensus.cc:2454] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:29.807883 24765 raft_consensus.cc:2454] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:29.808207 24567 leader_election.cc:302] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:29.808394 25651 raft_consensus.cc:2781] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:29.808472 25651 raft_consensus.cc:482] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:29.808522 25651 raft_consensus.cc:3037] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.809340 25651 raft_consensus.cc:504] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.809695 25651 leader_election.cc:288] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:29.810037 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5340b3277654975bbbdfb4312b7c00d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:29.810022 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5340b3277654975bbbdfb4312b7c00d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:29.810138 24765 raft_consensus.cc:3037] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.810138 25034 raft_consensus.cc:3037] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.811008 24765 raft_consensus.cc:2454] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:29.811008 25034 raft_consensus.cc:2454] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:29.811326 24564 leader_election.cc:302] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:29.811481 25651 raft_consensus.cc:2781] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:29.811602 25651 raft_consensus.cc:686] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:29.811745 25651 consensus_queue.cc:237] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:29.813421 24482 catalog_manager.cc:5484] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:29.847812 25651 raft_consensus.cc:482] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:29.847923 25651 raft_consensus.cc:504] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.848286 25651 leader_election.cc:288] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:29.848683 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c34f634d6480a862eeb642aaee3cc" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:29.848695 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c34f634d6480a862eeb642aaee3cc" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:29.848807 24897 raft_consensus.cc:2454] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:29.848826 25034 raft_consensus.cc:2454] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:29.849113 24564 leader_election.cc:302] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:29.849350 25651 raft_consensus.cc:2781] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:29.849436 25651 raft_consensus.cc:482] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:29.849488 25651 raft_consensus.cc:3037] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.850327 25651 raft_consensus.cc:504] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.850663 25651 leader_election.cc:288] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:29.850942 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c34f634d6480a862eeb642aaee3cc" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:29.851020 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803c34f634d6480a862eeb642aaee3cc" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:29.851051 24897 raft_consensus.cc:3037] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.851157 25034 raft_consensus.cc:3037] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:29.851825 24897 raft_consensus.cc:2454] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:29.851894 25034 raft_consensus.cc:2454] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:29.852118 24567 leader_election.cc:302] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:29.852298 25651 raft_consensus.cc:2781] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:29.852404 25651 raft_consensus.cc:686] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:29.852540 25651 consensus_queue.cc:237] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:29.854228 24482 catalog_manager.cc:5484] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:29.969759 25971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:29.970006 25971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:29.973140 25971 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:58:30.058207 24765 raft_consensus.cc:1261] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.059073 24632 raft_consensus.cc:1261] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.059464 25034 raft_consensus.cc:1261] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.059556 25032 raft_consensus.cc:1261] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.059574 24897 raft_consensus.cc:1261] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.059958 25890 consensus_queue.cc:1035] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.060323 25651 consensus_queue.cc:1035] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.060578 25812 consensus_queue.cc:1035] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.061384 25890 consensus_queue.cc:1035] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.061587 25651 consensus_queue.cc:1035] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.061892 24765 raft_consensus.cc:1261] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.066283 25812 consensus_queue.cc:1035] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.088375 25935 mvcc.cc:205] Tried to move back new op lower bound from 7023063490798878720 to 7023063489790193664. Current Snapshot: MvccSnapshot[applied={T|T < 7023063490798878720}]
I20240502 01:58:30.089006 25863 mvcc.cc:205] Tried to move back new op lower bound from 7023063490798878720 to 7023063489790193664. Current Snapshot: MvccSnapshot[applied={T|T < 7023063490798878720}]
INSERT report
rows total: 2000
time total: 119.1 ms
time per row: 0.0595498 ms
I20240502 01:58:30.130355 24484 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:30.135423 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 26b7501639064035a570b3b4c1cf083f (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.135560 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 26b7501639064035a570b3b4c1cf083f (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.135739 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7501639064035a570b3b4c1cf083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.135861 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7501639064035a570b3b4c1cf083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.136036 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 26b7501639064035a570b3b4c1cf083f (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.136142 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5f6bcbe0a048eb85b9475348b155ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.136270 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7501639064035a570b3b4c1cf083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.136363 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5f6bcbe0a048eb85b9475348b155ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.138118 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5f6bcbe0a048eb85b9475348b155ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.138320 25957 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:30.138418 25956 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:30.138897 25958 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:30.138852 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 70d0c3a7075a4e40a621b73931a3a33b (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.138852 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 70d0c3a7075a4e40a621b73931a3a33b (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.139065 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c3a7075a4e40a621b73931a3a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.139237 25956 tablet_bootstrap.cc:654] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.139688 25958 tablet_bootstrap.cc:654] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.138320 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5f6bcbe0a048eb85b9475348b155ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.140034 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c3a7075a4e40a621b73931a3a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.140515 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 70d0c3a7075a4e40a621b73931a3a33b (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.140769 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c3a7075a4e40a621b73931a3a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.142294 25957 tablet_bootstrap.cc:654] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.142287 25958 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:30.142361 25958 ts_tablet_manager.cc:1367] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:30.142567 25955 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:30.142755 25958 raft_consensus.cc:348] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.142868 25958 raft_consensus.cc:374] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.142897 25958 raft_consensus.cc:729] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.143013 25958 consensus_queue.cc:260] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.143291 25958 ts_tablet_manager.cc:1398] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.143369 25955 tablet_bootstrap.cc:654] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.143463 25956 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:30.143538 25956 ts_tablet_manager.cc:1367] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:30.143939 25956 raft_consensus.cc:348] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.144039 25956 raft_consensus.cc:374] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.144069 25956 raft_consensus.cc:729] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.144176 25956 consensus_queue.cc:260] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.144398 25956 ts_tablet_manager.cc:1398] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.144526 25956 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:30.144546 25955 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:30.144619 25955 ts_tablet_manager.cc:1367] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.144709 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5f6bcbe0a048eb85b9475348b155ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.144909 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5f6bcbe0a048eb85b9475348b155ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.145028 25955 raft_consensus.cc:348] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.145117 25955 raft_consensus.cc:374] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.145146 25955 raft_consensus.cc:729] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.145254 25955 consensus_queue.cc:260] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.145344 25956 tablet_bootstrap.cc:654] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.145486 25955 ts_tablet_manager.cc:1398] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.146369 25956 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:30.146440 25956 ts_tablet_manager.cc:1367] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.146667 25958 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:30.146786 25956 raft_consensus.cc:348] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.146864 25956 raft_consensus.cc:374] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.146893 25956 raft_consensus.cc:729] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.147008 25956 consensus_queue.cc:260] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.147253 25956 ts_tablet_manager.cc:1398] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.147392 25956 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:30.147508 25958 tablet_bootstrap.cc:654] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.147938 25957 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:30.148017 25957 ts_tablet_manager.cc:1367] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:30.148178 25956 tablet_bootstrap.cc:654] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.148406 25957 raft_consensus.cc:348] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.148479 25957 raft_consensus.cc:374] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.148505 25957 raft_consensus.cc:729] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.148597 25957 consensus_queue.cc:260] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.148847 25957 ts_tablet_manager.cc:1398] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.148916 25958 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:30.148988 25958 ts_tablet_manager.cc:1367] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:30.148990 25957 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:30.149380 25958 raft_consensus.cc:348] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.149479 25958 raft_consensus.cc:374] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.149510 25958 raft_consensus.cc:729] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.149613 25958 consensus_queue.cc:260] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.149724 25957 tablet_bootstrap.cc:654] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.149860 25958 ts_tablet_manager.cc:1398] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.150499 25956 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:30.150588 25956 ts_tablet_manager.cc:1367] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:30.150873 25957 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:30.150951 25957 ts_tablet_manager.cc:1367] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:30.150970 25956 raft_consensus.cc:348] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.151059 25956 raft_consensus.cc:374] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.151089 25956 raft_consensus.cc:729] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.151206 25956 consensus_queue.cc:260] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.151371 25957 raft_consensus.cc:348] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.151459 25956 ts_tablet_manager.cc:1398] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.151464 25957 raft_consensus.cc:374] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.151506 25957 raft_consensus.cc:729] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.151618 25957 consensus_queue.cc:260] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.151846 25957 ts_tablet_manager.cc:1398] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.151991 25957 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:30.152853 25957 tablet_bootstrap.cc:654] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.154296 25957 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:30.154384 25957 ts_tablet_manager.cc:1367] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.154836 25957 raft_consensus.cc:348] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.154947 25957 raft_consensus.cc:374] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.154980 25957 raft_consensus.cc:729] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.155097 25957 consensus_queue.cc:260] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.155380 25957 ts_tablet_manager.cc:1398] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:30.353826 25644 raft_consensus.cc:482] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:30.353958 25644 raft_consensus.cc:504] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.354349 25644 leader_election.cc:288] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:30.354707 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c3a7075a4e40a621b73931a3a33b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:30.354707 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c3a7075a4e40a621b73931a3a33b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:30.354837 24897 raft_consensus.cc:2454] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:30.354837 24632 raft_consensus.cc:2454] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:30.355082 24700 leader_election.cc:302] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:30.355268 25644 raft_consensus.cc:2781] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:30.355324 25644 raft_consensus.cc:482] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:30.355351 25644 raft_consensus.cc:3037] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.356127 25644 raft_consensus.cc:504] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.356395 25644 leader_election.cc:288] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:30.356705 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c3a7075a4e40a621b73931a3a33b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:30.356721 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d0c3a7075a4e40a621b73931a3a33b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:30.356820 24632 raft_consensus.cc:3037] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.356820 24897 raft_consensus.cc:3037] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.357095 25865 raft_consensus.cc:482] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:30.357205 25865 raft_consensus.cc:504] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.357560 25865 leader_election.cc:288] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:30.357726 24632 raft_consensus.cc:2454] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:30.357726 24897 raft_consensus.cc:2454] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:30.357818 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:30.357967 24898 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:30.358057 24701 leader_election.cc:302] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.358206 25644 raft_consensus.cc:2781] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:30.358290 24966 leader_election.cc:302] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.358345 25644 raft_consensus.cc:686] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:30.358394 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:30.358505 24764 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:30.358472 25644 consensus_queue.cc:237] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.358562 25865 raft_consensus.cc:2781] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:30.358603 25865 raft_consensus.cc:482] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:30.358634 25865 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.359522 25865 raft_consensus.cc:504] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.360070 25865 leader_election.cc:288] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:30.360077 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:30.360157 24898 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.360309 24482 catalog_manager.cc:5484] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:30.360433 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:30.360528 24764 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.360877 24898 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:30.361143 24966 leader_election.cc:302] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.361186 26006 raft_consensus.cc:482] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:30.361254 26006 raft_consensus.cc:504] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.361313 25865 raft_consensus.cc:2781] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:30.361466 25865 raft_consensus.cc:686] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:30.361614 26006 leader_election.cc:288] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.361593 25865 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:30.362536 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5f6bcbe0a048eb85b9475348b155ef" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:30.362646 24765 raft_consensus.cc:2454] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:30.362639 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5f6bcbe0a048eb85b9475348b155ef" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:30.362741 25032 raft_consensus.cc:2454] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:30.362875 24833 leader_election.cc:302] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.363032 26006 raft_consensus.cc:2781] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:30.363081 26006 raft_consensus.cc:482] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:30.363122 26006 raft_consensus.cc:3037] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.363530 24480 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:30.363970 26006 raft_consensus.cc:504] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.364276 26006 leader_election.cc:288] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.364588 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5f6bcbe0a048eb85b9475348b155ef" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:30.364704 24765 raft_consensus.cc:3037] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.364804 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5f6bcbe0a048eb85b9475348b155ef" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:30.364894 25032 raft_consensus.cc:3037] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.365556 25032 raft_consensus.cc:2454] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:30.365836 24830 leader_election.cc:302] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.366015 26006 raft_consensus.cc:2781] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:30.366096 26006 raft_consensus.cc:686] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:30.366206 26006 consensus_queue.cc:237] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.367600 24480 catalog_manager.cc:5484] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:30.382676 24765 raft_consensus.cc:2454] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:30.382808 24764 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
W20240502 01:58:30.487803 26021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:30.488008 26021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:30.490900 26021 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 01:58:30.573453 24898 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.573812 24632 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.574234 25644 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.574452 26016 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.575418 24764 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.576390 24897 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.576723 25865 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.580258 25865 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.584511 25032 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.584870 25790 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:30.586995 24765 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:30.589205 25928 mvcc.cc:205] Tried to move back new op lower bound from 7023063492905390080 to 7023063492029800448. Current Snapshot: MvccSnapshot[applied={T|T < 7023063492905390080}]
I20240502 01:58:30.591728 26052 mvcc.cc:205] Tried to move back new op lower bound from 7023063492905390080 to 7023063492029800448. Current Snapshot: MvccSnapshot[applied={T|T < 7023063492905390080}]
I20240502 01:58:30.592358 26006 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.241 ms
time per row: 0.0616204 ms
I20240502 01:58:30.652431 24480 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:30.657366 24611 tablet_service.cc:1467] Processing CreateTablet for tablet cf7bfd4c87a54b4fb31059ed03f955a2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.657474 24743 tablet_service.cc:1467] Processing CreateTablet for tablet cf7bfd4c87a54b4fb31059ed03f955a2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.657652 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7bfd4c87a54b4fb31059ed03f955a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.657734 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7bfd4c87a54b4fb31059ed03f955a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.658123 24745 tablet_service.cc:1467] Processing CreateTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.658152 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.658330 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6184a46a0d438da3bf1f3427a8c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.658349 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b2f9649fd4c7c9fbec0999f958e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.659531 26070 tablet_bootstrap.cc:492] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:30.660306 26070 tablet_bootstrap.cc:654] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.660606 25011 tablet_service.cc:1467] Processing CreateTablet for tablet cf7bfd4c87a54b4fb31059ed03f955a2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.660806 26071 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:30.660792 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.660852 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7bfd4c87a54b4fb31059ed03f955a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.660971 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b2f9649fd4c7c9fbec0999f958e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.661303 24610 tablet_service.cc:1467] Processing CreateTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.661470 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6184a46a0d438da3bf1f3427a8c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.661540 26071 tablet_bootstrap.cc:654] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.662189 26070 tablet_bootstrap.cc:492] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:30.662259 26070 ts_tablet_manager.cc:1367] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:30.662633 26070 raft_consensus.cc:348] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.662714 26070 raft_consensus.cc:374] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.662740 26070 raft_consensus.cc:729] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.662768 26071 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:30.662822 26074 tablet_bootstrap.cc:492] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:30.662842 26071 ts_tablet_manager.cc:1367] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:30.662868 26070 consensus_queue.cc:260] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.663218 26070 ts_tablet_manager.cc:1398] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.663240 26071 raft_consensus.cc:348] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.663404 26070 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:30.663406 26071 raft_consensus.cc:374] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.663445 26071 raft_consensus.cc:729] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.663621 26074 tablet_bootstrap.cc:654] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.663597 26071 consensus_queue.cc:260] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.663923 26071 ts_tablet_manager.cc:1398] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.663961 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.664152 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b2f9649fd4c7c9fbec0999f958e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.664556 26074 tablet_bootstrap.cc:492] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:30.664626 26074 ts_tablet_manager.cc:1367] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.665019 26074 raft_consensus.cc:348] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.665120 26074 raft_consensus.cc:374] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.665151 26074 raft_consensus.cc:729] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.665256 26074 consensus_queue.cc:260] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.665503 26074 ts_tablet_manager.cc:1398] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.665799 26074 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:30.666203 26070 tablet_bootstrap.cc:654] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.666605 26074 tablet_bootstrap.cc:654] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.667191 25011 tablet_service.cc:1467] Processing CreateTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:30.667400 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6184a46a0d438da3bf1f3427a8c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:30.667526 26074 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:30.667600 26074 ts_tablet_manager.cc:1367] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.667954 26074 raft_consensus.cc:348] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.668048 26074 raft_consensus.cc:374] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.668107 26074 raft_consensus.cc:729] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.668215 26074 consensus_queue.cc:260] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.668444 26074 ts_tablet_manager.cc:1398] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.669313 26074 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:30.670140 26074 tablet_bootstrap.cc:654] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.670393 26070 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:30.670464 26070 ts_tablet_manager.cc:1367] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:58:30.670776 26070 raft_consensus.cc:348] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.670855 26070 raft_consensus.cc:374] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.670882 26070 raft_consensus.cc:729] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.670971 26074 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:30.670996 26070 consensus_queue.cc:260] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.671049 26074 ts_tablet_manager.cc:1367] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.671229 26070 ts_tablet_manager.cc:1398] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.671694 26074 raft_consensus.cc:348] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.671842 26074 raft_consensus.cc:374] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.671880 26074 raft_consensus.cc:729] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.671975 26074 consensus_queue.cc:260] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.672186 26074 ts_tablet_manager.cc:1398] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.675566 26079 tablet_bootstrap.cc:492] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:30.676409 26079 tablet_bootstrap.cc:654] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.677240 26079 tablet_bootstrap.cc:492] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:30.677307 26079 ts_tablet_manager.cc:1367] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.677604 26079 raft_consensus.cc:348] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.677683 26079 raft_consensus.cc:374] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.677713 26079 raft_consensus.cc:729] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.677825 26079 consensus_queue.cc:260] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.678078 26079 ts_tablet_manager.cc:1398] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:30.678239 26079 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:30.678882 26079 tablet_bootstrap.cc:654] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.679719 26079 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:30.679836 26079 ts_tablet_manager.cc:1367] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:30.680243 26079 raft_consensus.cc:348] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.680322 26079 raft_consensus.cc:374] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.680348 26079 raft_consensus.cc:729] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.680502 26079 consensus_queue.cc:260] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.680733 26079 ts_tablet_manager.cc:1398] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.680888 26079 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:30.681535 26079 tablet_bootstrap.cc:654] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:30.682947 26079 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:30.683086 26079 ts_tablet_manager.cc:1367] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:30.683564 26079 raft_consensus.cc:348] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.683687 26079 raft_consensus.cc:374] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:30.683735 26079 raft_consensus.cc:729] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.683836 26079 consensus_queue.cc:260] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.684078 26079 ts_tablet_manager.cc:1398] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.868196 25999 raft_consensus.cc:482] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:30.868326 25999 raft_consensus.cc:504] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.868714 25999 leader_election.cc:288] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.869158 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6184a46a0d438da3bf1f3427a8c890" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:30.869158 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6184a46a0d438da3bf1f3427a8c890" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:30.869321 24765 raft_consensus.cc:2454] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:30.869321 25032 raft_consensus.cc:2454] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:30.869637 24564 leader_election.cc:302] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:30.869838 25999 raft_consensus.cc:2781] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:30.869902 25999 raft_consensus.cc:482] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:30.869956 25999 raft_consensus.cc:3037] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.870792 25999 raft_consensus.cc:504] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.871098 25999 leader_election.cc:288] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.871439 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6184a46a0d438da3bf1f3427a8c890" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:30.871462 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6184a46a0d438da3bf1f3427a8c890" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:30.871567 25032 raft_consensus.cc:3037] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.871567 24765 raft_consensus.cc:3037] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.872505 25032 raft_consensus.cc:2454] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:30.872500 24765 raft_consensus.cc:2454] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:30.872797 24564 leader_election.cc:302] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:30.872969 25999 raft_consensus.cc:2781] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:30.873191 25999 raft_consensus.cc:686] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:30.873335 25999 consensus_queue.cc:237] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:30.875053 24480 catalog_manager.cc:5484] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:30.889684 26006 raft_consensus.cc:482] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:30.889779 26006 raft_consensus.cc:504] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.890134 26006 leader_election.cc:288] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.890465 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b2f9649fd4c7c9fbec0999f958e7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:30.890466 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b2f9649fd4c7c9fbec0999f958e7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:30.890586 25032 raft_consensus.cc:2454] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:30.890586 24632 raft_consensus.cc:2454] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:30.890846 24830 leader_election.cc:302] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.891013 26006 raft_consensus.cc:2781] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:30.891075 26006 raft_consensus.cc:482] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:30.891101 26006 raft_consensus.cc:3037] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.891803 26006 raft_consensus.cc:504] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.892086 26006 leader_election.cc:288] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.892489 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b2f9649fd4c7c9fbec0999f958e7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:30.892489 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b2f9649fd4c7c9fbec0999f958e7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:30.892604 24632 raft_consensus.cc:3037] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.892603 25032 raft_consensus.cc:3037] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.893445 25032 raft_consensus.cc:2454] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:30.893438 24632 raft_consensus.cc:2454] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:30.893709 24830 leader_election.cc:302] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:30.893882 26006 raft_consensus.cc:2781] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:30.894001 26006 raft_consensus.cc:686] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:30.894129 26006 consensus_queue.cc:237] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:30.895700 24480 catalog_manager.cc:5484] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:30.929141 26060 raft_consensus.cc:482] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:30.929235 26060 raft_consensus.cc:504] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.929589 26060 leader_election.cc:288] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.929883 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7bfd4c87a54b4fb31059ed03f955a2" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:30.929916 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7bfd4c87a54b4fb31059ed03f955a2" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:30.930033 25032 raft_consensus.cc:2454] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:30.930033 24632 raft_consensus.cc:2454] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:30.930269 24700 leader_election.cc:302] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:30.930496 26060 raft_consensus.cc:2781] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:30.930684 26060 raft_consensus.cc:482] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:30.930825 26060 raft_consensus.cc:3037] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.932708 26060 raft_consensus.cc:504] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.933084 26060 leader_election.cc:288] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:30.933476 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7bfd4c87a54b4fb31059ed03f955a2" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:30.933583 25032 raft_consensus.cc:3037] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.933593 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7bfd4c87a54b4fb31059ed03f955a2" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:30.933687 24632 raft_consensus.cc:3037] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:30.934437 25032 raft_consensus.cc:2454] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:30.934505 24632 raft_consensus.cc:2454] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:30.934720 24698 leader_election.cc:302] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:30.934921 26060 raft_consensus.cc:2781] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:30.935006 26060 raft_consensus.cc:686] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:30.935128 26060 consensus_queue.cc:237] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:30.936661 24480 catalog_manager.cc:5484] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:31.056057 26087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.056263 26087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.059185 26087 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 01:58:31.145529 25032 raft_consensus.cc:1261] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.145879 24765 raft_consensus.cc:1261] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.146479 26083 consensus_queue.cc:1035] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.148162 26083 consensus_queue.cc:1035] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.151476 25031 raft_consensus.cc:1261] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.152004 24632 raft_consensus.cc:1261] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.152557 25790 consensus_queue.cc:1035] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.152814 25790 consensus_queue.cc:1035] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.155469 25034 raft_consensus.cc:1261] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.155814 26060 consensus_queue.cc:1035] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.156577 24631 raft_consensus.cc:1261] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.158373 26060 consensus_queue.cc:1035] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.98 ms
time per row: 0.0584901 ms
I20240502 01:58:31.215757 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:31.220465 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac6a4310330499483182c93827343c0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.220561 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac6a4310330499483182c93827343c0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.220747 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac6a4310330499483182c93827343c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.220830 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac6a4310330499483182c93827343c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.221414 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac6a4310330499483182c93827343c0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.221500 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 69182e5f4ae7401f89d22f063fa8df63 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.221663 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac6a4310330499483182c93827343c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.221719 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69182e5f4ae7401f89d22f063fa8df63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.222959 26149 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:31.223773 26149 tablet_bootstrap.cc:654] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.223872 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 92382869887b43f9a23a2c0ea0012720 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.224088 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92382869887b43f9a23a2c0ea0012720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.224272 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 69182e5f4ae7401f89d22f063fa8df63 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.224473 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69182e5f4ae7401f89d22f063fa8df63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.225525 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 92382869887b43f9a23a2c0ea0012720 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.225731 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92382869887b43f9a23a2c0ea0012720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.226114 26151 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:31.226212 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 92382869887b43f9a23a2c0ea0012720 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.226433 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92382869887b43f9a23a2c0ea0012720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.226722 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 69182e5f4ae7401f89d22f063fa8df63 (DEFAULT_TABLE table=rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.226926 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69182e5f4ae7401f89d22f063fa8df63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.227155 26151 tablet_bootstrap.cc:654] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.228169 26149 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:31.228169 26151 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:31.228247 26151 ts_tablet_manager.cc:1367] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.228247 26149 ts_tablet_manager.cc:1367] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:31.228626 26151 raft_consensus.cc:348] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.228626 26149 raft_consensus.cc:348] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.228716 26151 raft_consensus.cc:374] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.228716 26149 raft_consensus.cc:374] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.228746 26151 raft_consensus.cc:729] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.228746 26149 raft_consensus.cc:729] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.228850 26151 consensus_queue.cc:260] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.228850 26149 consensus_queue.cc:260] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.229110 26149 ts_tablet_manager.cc:1398] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.229110 26151 ts_tablet_manager.cc:1398] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.229243 26151 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:31.229243 26149 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:31.229305 26150 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:31.229948 26153 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:31.229998 26151 tablet_bootstrap.cc:654] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.229998 26149 tablet_bootstrap.cc:654] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.230190 26150 tablet_bootstrap.cc:654] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.230791 26153 tablet_bootstrap.cc:654] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.230916 26151 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:31.230990 26151 ts_tablet_manager.cc:1367] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:31.231103 26150 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:31.231195 26150 ts_tablet_manager.cc:1367] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.231377 26151 raft_consensus.cc:348] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.231468 26151 raft_consensus.cc:374] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.231499 26151 raft_consensus.cc:729] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.231601 26151 consensus_queue.cc:260] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.231619 26150 raft_consensus.cc:348] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.231709 26150 raft_consensus.cc:374] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.231740 26150 raft_consensus.cc:729] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.231846 26151 ts_tablet_manager.cc:1398] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:31.231845 26150 consensus_queue.cc:260] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.232081 26153 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:31.232092 26150 ts_tablet_manager.cc:1398] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.232151 26153 ts_tablet_manager.cc:1367] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:31.232245 26150 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:31.232573 26153 raft_consensus.cc:348] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.232676 26153 raft_consensus.cc:374] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.232713 26153 raft_consensus.cc:729] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.232831 26153 consensus_queue.cc:260] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.233048 26150 tablet_bootstrap.cc:654] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.233103 26153 ts_tablet_manager.cc:1398] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.233249 26153 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:31.234042 26153 tablet_bootstrap.cc:654] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.234436 26150 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:31.234519 26150 ts_tablet_manager.cc:1367] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.234896 26150 raft_consensus.cc:348] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.234987 26150 raft_consensus.cc:374] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.235018 26150 raft_consensus.cc:729] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.235114 26150 consensus_queue.cc:260] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.235380 26150 ts_tablet_manager.cc:1398] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.235896 26149 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:31.235972 26149 ts_tablet_manager.cc:1367] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:58:31.236259 26153 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:31.236325 26153 ts_tablet_manager.cc:1367] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:31.236315 26149 raft_consensus.cc:348] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.236418 26149 raft_consensus.cc:374] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.236464 26149 raft_consensus.cc:729] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.236567 26149 consensus_queue.cc:260] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.236653 26153 raft_consensus.cc:348] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.236743 26153 raft_consensus.cc:374] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.236770 26153 raft_consensus.cc:729] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.236799 26149 ts_tablet_manager.cc:1398] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.236887 26153 consensus_queue.cc:260] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.236938 26149 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:31.237156 26153 ts_tablet_manager.cc:1398] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:31.237605 26149 tablet_bootstrap.cc:654] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.238984 26149 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:31.239068 26149 ts_tablet_manager.cc:1367] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:31.239454 26149 raft_consensus.cc:348] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.239557 26149 raft_consensus.cc:374] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.239589 26149 raft_consensus.cc:729] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.239701 26149 consensus_queue.cc:260] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.239938 26149 ts_tablet_manager.cc:1398] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.289672 26016 raft_consensus.cc:482] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:31.289765 26016 raft_consensus.cc:504] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.290150 26016 leader_election.cc:288] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:31.290486 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:31.290539 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:31.290616 25034 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:31.290668 24898 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:31.290899 24698 leader_election.cc:302] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:31.291061 26016 raft_consensus.cc:2781] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:31.291117 26016 raft_consensus.cc:482] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:31.291170 26016 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.291931 26016 raft_consensus.cc:504] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.292229 26016 leader_election.cc:288] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:31.292487 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:31.292517 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:31.292588 25034 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.292603 24898 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.293258 24898 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:31.293268 25034 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:31.293506 24701 leader_election.cc:302] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:31.293699 26016 raft_consensus.cc:2781] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:31.293768 26016 raft_consensus.cc:686] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:31.293884 26016 consensus_queue.cc:237] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.295363 24483 catalog_manager.cc:5484] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:31.541136 26142 raft_consensus.cc:482] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:31.541261 26142 raft_consensus.cc:504] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.541709 26142 leader_election.cc:288] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:31.542021 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69182e5f4ae7401f89d22f063fa8df63" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:31.542083 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69182e5f4ae7401f89d22f063fa8df63" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:31.542161 25032 raft_consensus.cc:2454] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:31.542215 24898 raft_consensus.cc:2454] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:31.542407 24564 leader_election.cc:302] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:31.542593 26142 raft_consensus.cc:2781] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:31.542661 26142 raft_consensus.cc:482] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:31.542699 26142 raft_consensus.cc:3037] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.543502 26142 raft_consensus.cc:504] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.543794 26142 leader_election.cc:288] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:31.544111 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69182e5f4ae7401f89d22f063fa8df63" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:31.544220 25032 raft_consensus.cc:3037] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.544235 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69182e5f4ae7401f89d22f063fa8df63" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:31.544329 24898 raft_consensus.cc:3037] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.545132 25032 raft_consensus.cc:2454] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:31.545132 24898 raft_consensus.cc:2454] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:31.545413 24564 leader_election.cc:302] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:31.545567 26142 raft_consensus.cc:2781] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:31.545650 26142 raft_consensus.cc:686] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:31.545801 26142 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.547523 24483 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:31.568995 26016 raft_consensus.cc:482] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:31.569101 26016 raft_consensus.cc:504] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.569463 26016 leader_election.cc:288] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:31.569792 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac6a4310330499483182c93827343c0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:31.569831 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac6a4310330499483182c93827343c0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:31.569916 24898 raft_consensus.cc:2454] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:31.569985 24631 raft_consensus.cc:2454] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:31.570216 24701 leader_election.cc:302] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:31.570382 26016 raft_consensus.cc:2781] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:31.570436 26016 raft_consensus.cc:482] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:31.570473 26016 raft_consensus.cc:3037] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.571234 26016 raft_consensus.cc:504] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.571552 26016 leader_election.cc:288] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:31.571875 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac6a4310330499483182c93827343c0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:31.571977 24898 raft_consensus.cc:3037] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.572645 24898 raft_consensus.cc:2454] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:31.572865 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac6a4310330499483182c93827343c0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:31.572909 24701 leader_election.cc:302] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:31.572994 24631 raft_consensus.cc:3037] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.573779 24631 raft_consensus.cc:2454] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:31.573998 26016 raft_consensus.cc:2781] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:31.574086 26016 raft_consensus.cc:686] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:31.574218 26016 consensus_queue.cc:237] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.575991 24483 catalog_manager.cc:5484] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:31.692270 26166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.692476 26166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.695652 26166 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 01:58:31.742466 26060 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.754426 26016 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.805404 24897 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.805794 26060 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.810225 24631 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.811172 26016 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.828552 25032 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.829121 26142 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:31.834965 24898 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:31.835345 26142 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.011 ms
time per row: 0.0620056 ms
I20240502 01:58:31.864813 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:31.869805 24743 tablet_service.cc:1467] Processing CreateTablet for tablet a6846b4da6654802b00203529a01e398 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.870009 25012 tablet_service.cc:1467] Processing CreateTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.870175 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6846b4da6654802b00203529a01e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.870231 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfd63f6a0e84f3e855e7c07d3b9cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.870683 25011 tablet_service.cc:1467] Processing CreateTablet for tablet aafc084daeec44d5aa2ac9fca5615484 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.870901 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafc084daeec44d5aa2ac9fca5615484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.872402 26224 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:31.872577 24877 tablet_service.cc:1467] Processing CreateTablet for tablet a6846b4da6654802b00203529a01e398 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.872735 26223 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:31.872864 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6846b4da6654802b00203529a01e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.873499 26224 tablet_bootstrap.cc:654] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.873545 26223 tablet_bootstrap.cc:654] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.874583 26224 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:31.874961 24612 tablet_service.cc:1467] Processing CreateTablet for tablet a6846b4da6654802b00203529a01e398 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.875030 24610 tablet_service.cc:1467] Processing CreateTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.875221 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6846b4da6654802b00203529a01e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.875511 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfd63f6a0e84f3e855e7c07d3b9cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.875697 24877 tablet_service.cc:1467] Processing CreateTablet for tablet aafc084daeec44d5aa2ac9fca5615484 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.875710 24611 tablet_service.cc:1467] Processing CreateTablet for tablet aafc084daeec44d5aa2ac9fca5615484 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.875892 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafc084daeec44d5aa2ac9fca5615484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.875906 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafc084daeec44d5aa2ac9fca5615484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.874673 26224 ts_tablet_manager.cc:1367] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:31.877733 24743 tablet_service.cc:1467] Processing CreateTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (DEFAULT_TABLE table=rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:31.877897 26227 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:31.878005 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfd63f6a0e84f3e855e7c07d3b9cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.878095 26224 raft_consensus.cc:348] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.878198 26224 raft_consensus.cc:374] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.878229 26224 raft_consensus.cc:729] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.878350 26224 consensus_queue.cc:260] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.878505 26228 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:31.878634 26224 ts_tablet_manager.cc:1398] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.874752 26223 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:31.878870 26223 ts_tablet_manager.cc:1367] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:31.879321 26223 raft_consensus.cc:348] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.879469 26223 raft_consensus.cc:374] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.879526 26223 raft_consensus.cc:729] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.879658 26223 consensus_queue.cc:260] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.879952 26223 ts_tablet_manager.cc:1398] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.880017 26228 tablet_bootstrap.cc:654] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.880263 26227 tablet_bootstrap.cc:654] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.880589 26223 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:31.881050 26228 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:31.881153 26228 ts_tablet_manager.cc:1367] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:31.881611 26228 raft_consensus.cc:348] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.881664 26224 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:31.881718 26228 raft_consensus.cc:374] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.881760 26228 raft_consensus.cc:729] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.881786 26223 tablet_bootstrap.cc:654] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.881880 26228 consensus_queue.cc:260] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.882174 26228 ts_tablet_manager.cc:1398] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.882342 26228 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:31.882843 26224 tablet_bootstrap.cc:654] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.883121 26228 tablet_bootstrap.cc:654] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.883744 26224 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:31.883744 26227 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:31.883821 26227 ts_tablet_manager.cc:1367] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:31.883821 26224 ts_tablet_manager.cc:1367] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.884229 26227 raft_consensus.cc:348] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.884219 26224 raft_consensus.cc:348] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.884331 26227 raft_consensus.cc:374] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.884331 26224 raft_consensus.cc:374] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.884362 26227 raft_consensus.cc:729] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.884362 26224 raft_consensus.cc:729] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.884380 26228 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:31.884457 26228 ts_tablet_manager.cc:1367] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.884482 26224 consensus_queue.cc:260] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.884760 26224 ts_tablet_manager.cc:1398] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.884830 26228 raft_consensus.cc:348] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.884917 26228 raft_consensus.cc:374] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.884948 26228 raft_consensus.cc:729] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.885087 26228 consensus_queue.cc:260] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.885365 26228 ts_tablet_manager.cc:1398] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:31.885527 26228 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:31.886400 26228 tablet_bootstrap.cc:654] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.886358 26227 consensus_queue.cc:260] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.887073 26227 ts_tablet_manager.cc:1398] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:31.887233 26227 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:31.887472 26223 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:31.887557 26223 ts_tablet_manager.cc:1367] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:31.888041 26227 tablet_bootstrap.cc:654] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.887982 26223 raft_consensus.cc:348] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.888079 26223 raft_consensus.cc:374] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.888111 26223 raft_consensus.cc:729] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.888229 26223 consensus_queue.cc:260] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.888504 26223 ts_tablet_manager.cc:1398] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.889144 26228 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:31.889221 26228 ts_tablet_manager.cc:1367] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:31.889583 26228 raft_consensus.cc:348] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.889680 26228 raft_consensus.cc:374] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.889714 26228 raft_consensus.cc:729] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.889830 26228 consensus_queue.cc:260] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.890107 26228 ts_tablet_manager.cc:1398] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:31.890657 26227 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:31.890740 26227 ts_tablet_manager.cc:1367] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:31.891185 26227 raft_consensus.cc:348] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.891279 26227 raft_consensus.cc:374] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.891312 26227 raft_consensus.cc:729] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.891429 26227 consensus_queue.cc:260] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:31.891691 26227 ts_tablet_manager.cc:1398] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:31.937609 26131 raft_consensus.cc:482] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:31.937705 26131 raft_consensus.cc:504] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.938087 26131 leader_election.cc:288] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:31.938403 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfd63f6a0e84f3e855e7c07d3b9cc70" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:31.938447 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfd63f6a0e84f3e855e7c07d3b9cc70" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:31.938534 24631 raft_consensus.cc:2454] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:31.938585 24765 raft_consensus.cc:2454] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:31.938833 24965 leader_election.cc:302] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:31.938990 26131 raft_consensus.cc:2781] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:31.939059 26131 raft_consensus.cc:482] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:31.939105 26131 raft_consensus.cc:3037] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.939932 26131 raft_consensus.cc:504] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.940243 26131 leader_election.cc:288] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:31.940516 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfd63f6a0e84f3e855e7c07d3b9cc70" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:31.940570 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfd63f6a0e84f3e855e7c07d3b9cc70" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:31.940627 24631 raft_consensus.cc:3037] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.940682 24765 raft_consensus.cc:3037] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.941315 24631 raft_consensus.cc:2454] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:31.941408 24765 raft_consensus.cc:2454] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:31.941576 24965 leader_election.cc:302] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:31.941784 26131 raft_consensus.cc:2781] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:31.942000 26131 raft_consensus.cc:686] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:31.942131 26131 consensus_queue.cc:237] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:31.943588 24482 catalog_manager.cc:5484] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:31.965587 25999 raft_consensus.cc:482] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:31.965667 25999 raft_consensus.cc:504] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.965986 25999 leader_election.cc:288] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:31.966277 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6846b4da6654802b00203529a01e398" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:31.966384 24765 raft_consensus.cc:2454] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:31.966373 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6846b4da6654802b00203529a01e398" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:31.966516 24898 raft_consensus.cc:2454] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:31.966643 24567 leader_election.cc:302] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:31.966799 25999 raft_consensus.cc:2781] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:31.966854 25999 raft_consensus.cc:482] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:31.966898 25999 raft_consensus.cc:3037] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.967623 25999 raft_consensus.cc:504] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.967922 25999 leader_election.cc:288] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:31.968187 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6846b4da6654802b00203529a01e398" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:31.968281 24765 raft_consensus.cc:3037] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.968271 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6846b4da6654802b00203529a01e398" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:31.968371 24898 raft_consensus.cc:3037] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.969040 24765 raft_consensus.cc:2454] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:31.969107 24898 raft_consensus.cc:2454] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:31.969295 24567 leader_election.cc:302] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:31.969439 25999 raft_consensus.cc:2781] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:31.969527 25999 raft_consensus.cc:686] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:31.969659 25999 consensus_queue.cc:237] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:31.971246 24482 catalog_manager.cc:5484] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:32.147971 25999 raft_consensus.cc:482] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:32.148088 25999 raft_consensus.cc:504] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.148502 25999 leader_election.cc:288] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:32.148869 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafc084daeec44d5aa2ac9fca5615484" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:32.148869 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafc084daeec44d5aa2ac9fca5615484" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:32.149008 24898 raft_consensus.cc:2454] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:32.149008 25032 raft_consensus.cc:2454] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:32.149263 24567 leader_election.cc:302] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:32.149544 25999 raft_consensus.cc:2781] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:32.149641 25999 raft_consensus.cc:482] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:32.149694 25999 raft_consensus.cc:3037] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.150581 25999 raft_consensus.cc:504] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.150913 25999 leader_election.cc:288] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:32.151197 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafc084daeec44d5aa2ac9fca5615484" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:32.151250 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafc084daeec44d5aa2ac9fca5615484" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:32.151307 25032 raft_consensus.cc:3037] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.151340 24898 raft_consensus.cc:3037] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.152207 24898 raft_consensus.cc:2454] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:32.152207 25032 raft_consensus.cc:2454] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:32.152503 24564 leader_election.cc:302] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:32.152688 25999 raft_consensus.cc:2781] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:32.152769 25999 raft_consensus.cc:686] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:32.152901 25999 consensus_queue.cc:237] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.154716 24482 catalog_manager.cc:5484] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:32.281061 26240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:32.281240 26240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:32.284193 26240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 01:58:32.362205 24898 raft_consensus.cc:1261] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:32.362630 24765 raft_consensus.cc:1261] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:32.362649 26142 consensus_queue.cc:1035] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.363283 26142 consensus_queue.cc:1035] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.368026 24897 raft_consensus.cc:1261] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:32.368403 24764 raft_consensus.cc:1261] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:32.368405 25999 consensus_queue.cc:1035] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.368731 25032 raft_consensus.cc:1261] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:32.369573 26131 consensus_queue.cc:1035] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.370360 26142 consensus_queue.cc:1035] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.373199 24631 raft_consensus.cc:1261] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:32.374022 26131 consensus_queue.cc:1035] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.658 ms
time per row: 0.0593289 ms
I20240502 01:58:32.440258 24485 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:32.445394 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 89580549c13d4c1ebdb2a5d24da2a01e (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.445713 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89580549c13d4c1ebdb2a5d24da2a01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.446252 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 938581084b27405cbc01917e1a982208 (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.446475 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938581084b27405cbc01917e1a982208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.446408 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 938581084b27405cbc01917e1a982208 (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.446679 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938581084b27405cbc01917e1a982208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.447937 26299 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:32.448525 26300 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:32.448628 24611 tablet_service.cc:1467] Processing CreateTablet for tablet e720d8363bb049938798ab0b9aabc28d (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.448788 26299 tablet_bootstrap.cc:654] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.448882 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e720d8363bb049938798ab0b9aabc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.448499 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 89580549c13d4c1ebdb2a5d24da2a01e (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.449178 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89580549c13d4c1ebdb2a5d24da2a01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.449741 26300 tablet_bootstrap.cc:654] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.450346 24875 tablet_service.cc:1467] Processing CreateTablet for tablet e720d8363bb049938798ab0b9aabc28d (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.450573 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e720d8363bb049938798ab0b9aabc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.450875 26303 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:32.451745 26303 tablet_bootstrap.cc:654] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.452447 26299 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:32.452536 26299 ts_tablet_manager.cc:1367] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:32.452773 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 89580549c13d4c1ebdb2a5d24da2a01e (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.452898 26299 raft_consensus.cc:348] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.453001 26299 raft_consensus.cc:374] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.453020 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89580549c13d4c1ebdb2a5d24da2a01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.453037 26299 raft_consensus.cc:729] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.453078 24745 tablet_service.cc:1467] Processing CreateTablet for tablet e720d8363bb049938798ab0b9aabc28d (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.453169 26299 consensus_queue.cc:260] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.453312 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e720d8363bb049938798ab0b9aabc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.453507 26299 ts_tablet_manager.cc:1398] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:32.453663 26299 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:32.454428 26299 tablet_bootstrap.cc:654] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.455320 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 938581084b27405cbc01917e1a982208 (DEFAULT_TABLE table=rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.455391 26299 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:32.455475 26299 ts_tablet_manager.cc:1367] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:32.455547 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938581084b27405cbc01917e1a982208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.455883 26299 raft_consensus.cc:348] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.455976 26299 raft_consensus.cc:374] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.456008 26299 raft_consensus.cc:729] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.456120 26299 consensus_queue.cc:260] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.456465 26299 ts_tablet_manager.cc:1398] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:32.456552 26300 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:32.456614 26299 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:32.456624 26300 ts_tablet_manager.cc:1367] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240502 01:58:32.457190 26300 raft_consensus.cc:348] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.457279 26300 raft_consensus.cc:374] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.457309 26300 raft_consensus.cc:729] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.457386 26299 tablet_bootstrap.cc:654] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.457409 26300 consensus_queue.cc:260] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.457676 26300 ts_tablet_manager.cc:1398] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:32.457837 26300 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:32.458046 26308 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:32.458797 26308 tablet_bootstrap.cc:654] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.458932 26303 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:32.459008 26303 ts_tablet_manager.cc:1367] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:58:32.459466 26303 raft_consensus.cc:348] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.459579 26303 raft_consensus.cc:374] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.459619 26303 raft_consensus.cc:729] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.459736 26303 consensus_queue.cc:260] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.459800 26300 tablet_bootstrap.cc:654] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.459806 26299 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:32.459872 26299 ts_tablet_manager.cc:1367] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:32.460014 26303 ts_tablet_manager.cc:1398] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:32.460238 26299 raft_consensus.cc:348] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.460338 26299 raft_consensus.cc:374] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.460372 26299 raft_consensus.cc:729] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.460474 26299 consensus_queue.cc:260] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.460687 26299 ts_tablet_manager.cc:1398] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:32.462357 26300 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:32.462456 26300 ts_tablet_manager.cc:1367] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:32.462864 26300 raft_consensus.cc:348] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.462987 26300 raft_consensus.cc:374] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.463022 26300 raft_consensus.cc:729] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.463141 26300 consensus_queue.cc:260] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.463557 26300 ts_tablet_manager.cc:1398] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:32.463760 26300 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:32.464541 26300 tablet_bootstrap.cc:654] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.465447 26300 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:32.465456 26308 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:32.465525 26308 ts_tablet_manager.cc:1367] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:58:32.465523 26300 ts_tablet_manager.cc:1367] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:32.465907 26300 raft_consensus.cc:348] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.465907 26308 raft_consensus.cc:348] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.466029 26300 raft_consensus.cc:374] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.466029 26308 raft_consensus.cc:374] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.466059 26300 raft_consensus.cc:729] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.466059 26308 raft_consensus.cc:729] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.466168 26300 consensus_queue.cc:260] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.466169 26308 consensus_queue.cc:260] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.466411 26300 ts_tablet_manager.cc:1398] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:32.466413 26308 ts_tablet_manager.cc:1398] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:32.466570 26308 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:32.467278 26308 tablet_bootstrap.cc:654] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.468166 26308 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:32.468238 26308 ts_tablet_manager.cc:1367] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:32.468628 26308 raft_consensus.cc:348] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.468715 26308 raft_consensus.cc:374] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.468744 26308 raft_consensus.cc:729] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.468851 26308 consensus_queue.cc:260] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.469146 26308 ts_tablet_manager.cc:1398] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 01:58:32.668156 26193 raft_consensus.cc:482] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:32.668268 26193 raft_consensus.cc:504] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.668633 26193 leader_election.cc:288] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:32.668936 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938581084b27405cbc01917e1a982208" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:32.668987 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938581084b27405cbc01917e1a982208" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:32.669064 24764 raft_consensus.cc:2454] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:32.669121 25032 raft_consensus.cc:2454] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:32.669382 24833 leader_election.cc:302] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:32.669580 26193 raft_consensus.cc:2781] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:32.669642 26193 raft_consensus.cc:482] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:32.669685 26193 raft_consensus.cc:3037] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.670653 26193 raft_consensus.cc:504] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.670948 26193 leader_election.cc:288] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:32.671263 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938581084b27405cbc01917e1a982208" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:32.671263 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938581084b27405cbc01917e1a982208" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:32.671375 25032 raft_consensus.cc:3037] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.671375 24764 raft_consensus.cc:3037] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.672235 25032 raft_consensus.cc:2454] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:32.672235 24764 raft_consensus.cc:2454] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:32.672513 24833 leader_election.cc:302] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:32.672667 26193 raft_consensus.cc:2781] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:32.672878 26193 raft_consensus.cc:686] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:32.673019 26193 consensus_queue.cc:237] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.674810 24482 catalog_manager.cc:5484] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:32.681639 25999 raft_consensus.cc:482] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:32.681725 25999 raft_consensus.cc:504] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.682044 25999 leader_election.cc:288] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:32.682299 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e720d8363bb049938798ab0b9aabc28d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:32.682375 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e720d8363bb049938798ab0b9aabc28d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:32.682417 24764 raft_consensus.cc:2454] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:32.682493 24897 raft_consensus.cc:2454] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:32.682706 24567 leader_election.cc:302] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:32.682853 25999 raft_consensus.cc:2781] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:32.682915 25999 raft_consensus.cc:482] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:32.682960 25999 raft_consensus.cc:3037] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.683717 25999 raft_consensus.cc:504] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.684008 25999 leader_election.cc:288] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:32.684288 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e720d8363bb049938798ab0b9aabc28d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:32.684341 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e720d8363bb049938798ab0b9aabc28d" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:32.684381 24764 raft_consensus.cc:3037] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.684437 24897 raft_consensus.cc:3037] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.685254 24897 raft_consensus.cc:2454] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:32.685300 24764 raft_consensus.cc:2454] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:32.685513 24567 leader_election.cc:302] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:32.685758 25999 raft_consensus.cc:2781] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:32.685855 25999 raft_consensus.cc:686] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:32.686003 25999 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:32.687462 24482 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:32.754802 26193 raft_consensus.cc:482] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:32.754901 26193 raft_consensus.cc:504] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.755251 26193 leader_election.cc:288] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:32.755589 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:32.755589 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:32.755723 24765 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:32.755723 25032 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:32.755975 24830 leader_election.cc:302] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:32.756135 26193 raft_consensus.cc:2781] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:32.756193 26193 raft_consensus.cc:482] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:32.756232 26193 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.757167 26193 raft_consensus.cc:504] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.757448 26193 leader_election.cc:288] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:32.757700 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:32.757795 24765 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.758520 24765 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:32.758721 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:32.758773 24833 leader_election.cc:302] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:32.758826 25032 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.758924 26193 raft_consensus.cc:2781] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:32.759050 26193 raft_consensus.cc:686] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:32.759328 26193 consensus_queue.cc:237] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:32.759716 25032 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:32.761265 24482 catalog_manager.cc:5484] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:32.916738 26319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:32.917004 26319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:32.920044 26319 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 01:58:33.005977 24897 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.006385 25999 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.006681 25032 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.007519 26315 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.008229 24765 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.008966 25999 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.009119 24765 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.009498 26193 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.012182 25034 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.013485 26315 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.017222 24763 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.019654 26193 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.036163 26363 mvcc.cc:205] Tried to move back new op lower bound from 7023063502870405120 to 7023063501863596032. Current Snapshot: MvccSnapshot[applied={T|T < 7023063502870405120}]
INSERT report
rows total: 2000
time total: 115.971 ms
time per row: 0.0579853 ms
I20240502 01:58:33.076119 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:33.081156 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 2af8ec8342a9465e8af091c71c30599b (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.081209 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 2af8ec8342a9465e8af091c71c30599b (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.081326 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 2af8ec8342a9465e8af091c71c30599b (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.081440 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af8ec8342a9465e8af091c71c30599b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.081466 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af8ec8342a9465e8af091c71c30599b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.081595 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af8ec8342a9465e8af091c71c30599b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.083627 26380 tablet_bootstrap.cc:492] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:33.083750 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 0345ae8459544e049edf0ef0bcc5136d (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.083974 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345ae8459544e049edf0ef0bcc5136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.083920 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 0345ae8459544e049edf0ef0bcc5136d (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.084156 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345ae8459544e049edf0ef0bcc5136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.084359 26380 tablet_bootstrap.cc:654] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.084908 26379 tablet_bootstrap.cc:492] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:33.085589 26379 tablet_bootstrap.cc:654] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.086144 24745 tablet_service.cc:1467] Processing CreateTablet for tablet c7b1b5f0738c4e59a248ea7f946885a2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.086352 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b1b5f0738c4e59a248ea7f946885a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.086385 26378 tablet_bootstrap.cc:492] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:33.086805 24610 tablet_service.cc:1467] Processing CreateTablet for tablet c7b1b5f0738c4e59a248ea7f946885a2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.087008 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b1b5f0738c4e59a248ea7f946885a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.087100 26382 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:33.087342 26380 tablet_bootstrap.cc:492] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:33.087419 26380 ts_tablet_manager.cc:1367] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:33.087857 26382 tablet_bootstrap.cc:654] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.087829 26380 raft_consensus.cc:348] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.087921 26380 raft_consensus.cc:374] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.087950 26380 raft_consensus.cc:729] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.088088 26380 consensus_queue.cc:260] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.088356 26380 ts_tablet_manager.cc:1398] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.088352 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 0345ae8459544e049edf0ef0bcc5136d (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.088436 25012 tablet_service.cc:1467] Processing CreateTablet for tablet c7b1b5f0738c4e59a248ea7f946885a2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.088559 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345ae8459544e049edf0ef0bcc5136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.088976 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b1b5f0738c4e59a248ea7f946885a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.088984 26380 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:33.089748 26380 tablet_bootstrap.cc:654] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.090803 26379 tablet_bootstrap.cc:492] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:33.090932 26379 ts_tablet_manager.cc:1367] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:33.091348 26379 raft_consensus.cc:348] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.091526 26379 raft_consensus.cc:374] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.091531 26382 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:33.091614 26382 ts_tablet_manager.cc:1367] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:33.091590 26379 raft_consensus.cc:729] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.092034 26382 raft_consensus.cc:348] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.092144 26382 raft_consensus.cc:374] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.092187 26382 raft_consensus.cc:729] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.092288 26379 consensus_queue.cc:260] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.092329 26382 consensus_queue.cc:260] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.092624 26379 ts_tablet_manager.cc:1398] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:33.093034 26379 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:33.093176 26382 ts_tablet_manager.cc:1398] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:33.094110 26378 tablet_bootstrap.cc:654] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.094677 26379 tablet_bootstrap.cc:654] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.095050 26378 tablet_bootstrap.cc:492] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:33.095140 26378 ts_tablet_manager.cc:1367] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:33.095562 26378 raft_consensus.cc:348] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.095669 26378 raft_consensus.cc:374] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.095700 26378 raft_consensus.cc:729] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.095803 26378 consensus_queue.cc:260] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.096038 26380 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:33.096058 26378 ts_tablet_manager.cc:1398] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.096125 26380 ts_tablet_manager.cc:1367] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:58:33.096222 26378 tablet_bootstrap.cc:492] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:33.096496 26380 raft_consensus.cc:348] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.096589 26380 raft_consensus.cc:374] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.096621 26380 raft_consensus.cc:729] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.096721 26380 consensus_queue.cc:260] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.097018 26378 tablet_bootstrap.cc:654] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.097054 26380 ts_tablet_manager.cc:1398] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:33.097239 26380 tablet_bootstrap.cc:492] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:33.098691 26378 tablet_bootstrap.cc:492] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:33.098771 26378 ts_tablet_manager.cc:1367] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:33.099148 26378 raft_consensus.cc:348] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.099244 26378 raft_consensus.cc:374] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.099283 26378 raft_consensus.cc:729] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.099401 26378 consensus_queue.cc:260] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.099632 26379 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:33.099668 26378 ts_tablet_manager.cc:1398] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.099784 26379 ts_tablet_manager.cc:1367] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:33.100170 26379 raft_consensus.cc:348] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.100250 26379 raft_consensus.cc:374] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.100278 26379 raft_consensus.cc:729] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.100412 26379 consensus_queue.cc:260] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.100715 26379 ts_tablet_manager.cc:1398] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.100904 26380 tablet_bootstrap.cc:654] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.101095 26379 tablet_bootstrap.cc:492] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:33.102509 26380 tablet_bootstrap.cc:492] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:33.102530 26379 tablet_bootstrap.cc:654] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.102594 26380 ts_tablet_manager.cc:1367] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:58:33.102949 26380 raft_consensus.cc:348] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.103027 26380 raft_consensus.cc:374] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.103055 26380 raft_consensus.cc:729] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.103176 26380 consensus_queue.cc:260] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.103417 26380 ts_tablet_manager.cc:1398] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:33.103587 26379 tablet_bootstrap.cc:492] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:33.103668 26379 ts_tablet_manager.cc:1367] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:33.104014 26379 raft_consensus.cc:348] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.104092 26379 raft_consensus.cc:374] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.104146 26379 raft_consensus.cc:729] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.104269 26379 consensus_queue.cc:260] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.104609 26379 ts_tablet_manager.cc:1398] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.144594 25999 raft_consensus.cc:482] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:33.144691 25999 raft_consensus.cc:504] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.145007 25999 leader_election.cc:288] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:33.145337 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b1b5f0738c4e59a248ea7f946885a2" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:33.145347 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b1b5f0738c4e59a248ea7f946885a2" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:33.145455 25032 raft_consensus.cc:2454] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:33.145476 24765 raft_consensus.cc:2454] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:33.145730 24564 leader_election.cc:302] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.145889 25999 raft_consensus.cc:2781] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:33.145973 25999 raft_consensus.cc:482] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:33.146121 25999 raft_consensus.cc:3037] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.146873 25999 raft_consensus.cc:504] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.147159 25999 leader_election.cc:288] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:33.147437 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b1b5f0738c4e59a248ea7f946885a2" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:33.147454 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b1b5f0738c4e59a248ea7f946885a2" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:33.147543 24765 raft_consensus.cc:3037] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.147549 25032 raft_consensus.cc:3037] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.148204 25032 raft_consensus.cc:2454] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:33.148216 24765 raft_consensus.cc:2454] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:33.148452 24564 leader_election.cc:302] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.148600 25999 raft_consensus.cc:2781] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.148686 25999 raft_consensus.cc:686] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:33.148831 25999 consensus_queue.cc:237] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:33.150466 24487 catalog_manager.cc:5484] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:33.217867 26365 raft_consensus.cc:482] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:33.218015 26365 raft_consensus.cc:504] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.218359 26365 leader_election.cc:288] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:33.218660 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af8ec8342a9465e8af091c71c30599b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:33.218670 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af8ec8342a9465e8af091c71c30599b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:33.218775 25032 raft_consensus.cc:2454] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:33.218792 24631 raft_consensus.cc:2454] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:33.219066 24698 leader_election.cc:302] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.219224 26365 raft_consensus.cc:2781] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:33.219285 26365 raft_consensus.cc:482] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:33.219319 26365 raft_consensus.cc:3037] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.220129 26365 raft_consensus.cc:504] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.220438 26365 leader_election.cc:288] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:33.220741 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af8ec8342a9465e8af091c71c30599b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:33.220770 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af8ec8342a9465e8af091c71c30599b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:33.220834 25032 raft_consensus.cc:3037] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.220853 24631 raft_consensus.cc:3037] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.221508 24631 raft_consensus.cc:2454] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:33.221544 25032 raft_consensus.cc:2454] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:33.221797 24698 leader_election.cc:302] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.222002 26365 raft_consensus.cc:2781] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.222101 26365 raft_consensus.cc:686] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:33.222239 26365 consensus_queue.cc:237] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.223907 24487 catalog_manager.cc:5484] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:33.235888 26391 raft_consensus.cc:482] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:33.235975 26391 raft_consensus.cc:504] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.236284 26391 leader_election.cc:288] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:33.236550 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345ae8459544e049edf0ef0bcc5136d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:33.236582 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345ae8459544e049edf0ef0bcc5136d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:33.236660 24631 raft_consensus.cc:2454] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:33.236688 24897 raft_consensus.cc:2454] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:33.236894 24965 leader_election.cc:302] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.237056 26391 raft_consensus.cc:2781] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:33.237113 26391 raft_consensus.cc:482] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:33.237152 26391 raft_consensus.cc:3037] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.237854 26391 raft_consensus.cc:504] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.238185 26391 leader_election.cc:288] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:33.238456 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345ae8459544e049edf0ef0bcc5136d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:33.238477 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0345ae8459544e049edf0ef0bcc5136d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:33.238548 24631 raft_consensus.cc:3037] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.238566 24897 raft_consensus.cc:3037] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.239341 24631 raft_consensus.cc:2454] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:33.239446 24897 raft_consensus.cc:2454] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:33.239634 24965 leader_election.cc:302] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.239777 26391 raft_consensus.cc:2781] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.239846 26391 raft_consensus.cc:686] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:33.239960 26391 consensus_queue.cc:237] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:33.241531 24487 catalog_manager.cc:5484] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:33.378904 26395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:33.379102 26395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:33.382103 26395 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 01:58:33.460546 25032 raft_consensus.cc:1261] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.460918 24765 raft_consensus.cc:1261] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.461519 26142 consensus_queue.cc:1035] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.462265 26142 consensus_queue.cc:1035] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.464694 24897 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.465165 26391 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.465411 24631 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.466877 26391 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.469058 26359 mvcc.cc:205] Tried to move back new op lower bound from 7023063504746029056 to 7023063503831965696. Current Snapshot: MvccSnapshot[applied={T|T < 7023063504746029056}]
I20240502 01:58:33.473603 26348 mvcc.cc:205] Tried to move back new op lower bound from 7023063504730144768 to 7023063503459020800. Current Snapshot: MvccSnapshot[applied={T|T < 7023063504730144768}]
I20240502 01:58:33.489547 25032 raft_consensus.cc:1261] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.489897 26365 consensus_queue.cc:1035] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.490553 24631 raft_consensus.cc:1261] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:33.490918 26365 consensus_queue.cc:1035] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.607 ms
time per row: 0.0628035 ms
I20240502 01:58:33.546188 24478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:33.551359 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 953a778a807a484b9e0376bdeeed8343 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.551577 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 953a778a807a484b9e0376bdeeed8343 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.551663 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953a778a807a484b9e0376bdeeed8343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.551828 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953a778a807a484b9e0376bdeeed8343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.551869 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8667a49f1a44339122040b0122be0d (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.552098 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8667a49f1a44339122040b0122be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.553802 26382 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:33.553985 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 953a778a807a484b9e0376bdeeed8343 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.554229 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953a778a807a484b9e0376bdeeed8343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.554577 26382 tablet_bootstrap.cc:654] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.555294 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8667a49f1a44339122040b0122be0d (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.555366 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 086c9b674a0543cab3c82349d151e373 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.555536 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8667a49f1a44339122040b0122be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.555665 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086c9b674a0543cab3c82349d151e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.555945 26378 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:33.556665 26378 tablet_bootstrap.cc:654] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.557194 26382 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:33.557277 26382 ts_tablet_manager.cc:1367] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:33.557685 26382 raft_consensus.cc:348] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.557780 26382 raft_consensus.cc:374] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.557807 26382 raft_consensus.cc:729] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.557914 26382 consensus_queue.cc:260] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.558218 26382 ts_tablet_manager.cc:1398] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:33.558250 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 086c9b674a0543cab3c82349d151e373 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.558391 26382 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:33.558445 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086c9b674a0543cab3c82349d151e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.559051 26382 tablet_bootstrap.cc:654] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.559306 26380 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:33.559425 26379 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:33.560065 26380 tablet_bootstrap.cc:654] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.560169 26379 tablet_bootstrap.cc:654] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.560715 26378 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:33.560798 26378 ts_tablet_manager.cc:1367] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:33.561208 26378 raft_consensus.cc:348] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.561314 26378 raft_consensus.cc:374] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.561344 26378 raft_consensus.cc:729] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.561350 26382 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:33.561462 26378 consensus_queue.cc:260] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.561651 26380 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:33.561632 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8667a49f1a44339122040b0122be0d (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.561718 26378 ts_tablet_manager.cc:1398] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.561726 26380 ts_tablet_manager.cc:1367] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:33.561841 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8667a49f1a44339122040b0122be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.561866 26378 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:33.562433 26380 raft_consensus.cc:348] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.562541 26380 raft_consensus.cc:374] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.562573 26380 raft_consensus.cc:729] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.562676 26378 tablet_bootstrap.cc:654] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.562671 26380 consensus_queue.cc:260] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.562786 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 086c9b674a0543cab3c82349d151e373 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:33.562918 26380 ts_tablet_manager.cc:1398] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.562994 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086c9b674a0543cab3c82349d151e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.563521 26382 ts_tablet_manager.cc:1367] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:33.563886 26382 raft_consensus.cc:348] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.563992 26382 raft_consensus.cc:374] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.564021 26382 raft_consensus.cc:729] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.564114 26382 consensus_queue.cc:260] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.564426 26382 ts_tablet_manager.cc:1398] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:33.564745 26382 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:33.565536 26382 tablet_bootstrap.cc:654] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.565595 26379 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:33.565670 26379 ts_tablet_manager.cc:1367] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:33.566102 26379 raft_consensus.cc:348] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.566205 26379 raft_consensus.cc:374] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.566234 26379 raft_consensus.cc:729] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.566349 26379 consensus_queue.cc:260] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.566627 26379 ts_tablet_manager.cc:1398] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.566756 26379 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:33.567512 26380 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:33.567576 26379 tablet_bootstrap.cc:654] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.569692 26379 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:33.569753 26379 ts_tablet_manager.cc:1367] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:33.570132 26379 raft_consensus.cc:348] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.570219 26379 raft_consensus.cc:374] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.570246 26379 raft_consensus.cc:729] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.570365 26379 consensus_queue.cc:260] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.570475 26378 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:33.570545 26378 ts_tablet_manager.cc:1367] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:58:33.570588 26379 ts_tablet_manager.cc:1398] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.570832 26378 raft_consensus.cc:348] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.570896 26378 raft_consensus.cc:374] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.570914 26378 raft_consensus.cc:729] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.571007 26378 consensus_queue.cc:260] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.571242 26378 ts_tablet_manager.cc:1398] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:33.575598 26380 tablet_bootstrap.cc:654] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.576660 26382 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:33.577124 26380 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:33.577134 26382 ts_tablet_manager.cc:1367] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:58:33.577199 26380 ts_tablet_manager.cc:1367] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240502 01:58:33.577565 26380 raft_consensus.cc:348] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.577661 26380 raft_consensus.cc:374] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.577692 26380 raft_consensus.cc:729] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.577644 26382 raft_consensus.cc:348] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.577735 26382 raft_consensus.cc:374] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.577771 26382 raft_consensus.cc:729] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.577797 26380 consensus_queue.cc:260] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.577876 26382 consensus_queue.cc:260] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.578088 26380 ts_tablet_manager.cc:1398] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.578577 26382 ts_tablet_manager.cc:1398] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:33.584199 26142 raft_consensus.cc:482] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:33.584268 26142 raft_consensus.cc:504] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.584563 26142 leader_election.cc:288] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:33.584818 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953a778a807a484b9e0376bdeeed8343" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:33.584879 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953a778a807a484b9e0376bdeeed8343" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:33.584937 24897 raft_consensus.cc:2454] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:33.585019 24765 raft_consensus.cc:2454] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:33.585193 24567 leader_election.cc:302] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:33.585350 26142 raft_consensus.cc:2781] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:33.585405 26142 raft_consensus.cc:482] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:33.585451 26142 raft_consensus.cc:3037] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.586248 26142 raft_consensus.cc:504] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.586524 26142 leader_election.cc:288] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:33.586804 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953a778a807a484b9e0376bdeeed8343" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:33.586853 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953a778a807a484b9e0376bdeeed8343" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:33.586905 24897 raft_consensus.cc:3037] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.586946 24765 raft_consensus.cc:3037] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.587594 24765 raft_consensus.cc:2454] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:33.587707 24897 raft_consensus.cc:2454] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:33.587961 24567 leader_election.cc:302] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:33.588110 26142 raft_consensus.cc:2781] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.588201 26142 raft_consensus.cc:686] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:33.588306 26142 consensus_queue.cc:237] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.590051 24478 catalog_manager.cc:5484] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:33.643430 26297 raft_consensus.cc:482] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:33.643536 26297 raft_consensus.cc:504] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.643869 26297 leader_election.cc:288] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:33.644176 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086c9b674a0543cab3c82349d151e373" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:33.644217 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086c9b674a0543cab3c82349d151e373" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:33.644289 24897 raft_consensus.cc:2454] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:33.644345 24765 raft_consensus.cc:2454] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:33.644587 24966 leader_election.cc:302] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.644733 26297 raft_consensus.cc:2781] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:33.644788 26297 raft_consensus.cc:482] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:33.644811 26297 raft_consensus.cc:3037] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.645633 26297 raft_consensus.cc:504] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.645922 26297 leader_election.cc:288] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:33.646252 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086c9b674a0543cab3c82349d151e373" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:33.646299 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086c9b674a0543cab3c82349d151e373" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:33.646347 24897 raft_consensus.cc:3037] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.646409 24765 raft_consensus.cc:3037] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.647037 24897 raft_consensus.cc:2454] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:33.647078 24765 raft_consensus.cc:2454] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:33.647285 24966 leader_election.cc:302] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:33.647437 26297 raft_consensus.cc:2781] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.647544 26297 raft_consensus.cc:686] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:33.647681 26297 consensus_queue.cc:237] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:33.649250 24478 catalog_manager.cc:5484] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:33.804529 26297 raft_consensus.cc:482] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:33.804672 26297 raft_consensus.cc:504] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.805135 26297 leader_election.cc:288] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:33.805608 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8667a49f1a44339122040b0122be0d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:33.805629 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8667a49f1a44339122040b0122be0d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:33.805750 24897 raft_consensus.cc:2454] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:33.805781 24631 raft_consensus.cc:2454] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:33.806119 24965 leader_election.cc:302] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.806326 26297 raft_consensus.cc:2781] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:33.806385 26297 raft_consensus.cc:482] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:33.806421 26297 raft_consensus.cc:3037] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.807304 26297 raft_consensus.cc:504] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.807627 26297 leader_election.cc:288] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:33.807931 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8667a49f1a44339122040b0122be0d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:33.807977 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8667a49f1a44339122040b0122be0d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:33.808040 24897 raft_consensus.cc:3037] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.808076 24631 raft_consensus.cc:3037] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.808775 24631 raft_consensus.cc:2454] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:33.808841 24897 raft_consensus.cc:2454] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:33.809072 24965 leader_election.cc:302] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:33.809244 26297 raft_consensus.cc:2781] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.809330 26297 raft_consensus.cc:686] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:33.809468 26297 consensus_queue.cc:237] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:33.811326 24478 catalog_manager.cc:5484] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:33.956322 26447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:33.956612 26447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:33.959822 26447 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 01:58:33.984378 26427 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:33.988036 26142 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.035795 24765 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.036180 26297 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.039666 24631 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.040028 26391 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.040341 24897 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.041587 26297 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.049260 24898 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.049607 26297 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.074843 26349 mvcc.cc:205] Tried to move back new op lower bound from 7023063507101605888 to 7023063506164740096. Current Snapshot: MvccSnapshot[applied={T|T < 7023063507101605888}]
INSERT report
rows total: 2000
time total: 103.104 ms
time per row: 0.0515519 ms
I20240502 01:58:34.106197 24480 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:34.111089 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.111169 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.111387 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ede6fc65d044804848e6f61e8c238c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.111433 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ede6fc65d044804848e6f61e8c238c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.111949 24612 tablet_service.cc:1467] Processing CreateTablet for tablet fa3e57ee511e4d9886fcfed3851e4c38 (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.112087 24877 tablet_service.cc:1467] Processing CreateTablet for tablet fa3e57ee511e4d9886fcfed3851e4c38 (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.112188 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3e57ee511e4d9886fcfed3851e4c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.112298 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3e57ee511e4d9886fcfed3851e4c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.113694 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.113902 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ede6fc65d044804848e6f61e8c238c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.114326 26504 tablet_bootstrap.cc:492] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:34.114701 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 9f999b09b79a4632936833f351c4b28b (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.114764 25012 tablet_service.cc:1467] Processing CreateTablet for tablet fa3e57ee511e4d9886fcfed3851e4c38 (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.114941 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f999b09b79a4632936833f351c4b28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.115170 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3e57ee511e4d9886fcfed3851e4c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.115821 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 9f999b09b79a4632936833f351c4b28b (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.116019 26504 tablet_bootstrap.cc:654] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.116039 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f999b09b79a4632936833f351c4b28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.117033 26506 tablet_bootstrap.cc:492] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:34.117475 26503 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:34.117651 26504 tablet_bootstrap.cc:492] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:34.117823 26506 tablet_bootstrap.cc:654] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.117956 26504 ts_tablet_manager.cc:1367] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:34.118245 26503 tablet_bootstrap.cc:654] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.118395 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 9f999b09b79a4632936833f351c4b28b (DEFAULT_TABLE table=rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.118391 26504 raft_consensus.cc:348] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.118487 26504 raft_consensus.cc:374] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.118516 26504 raft_consensus.cc:729] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.118616 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f999b09b79a4632936833f351c4b28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.118626 26504 consensus_queue.cc:260] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.118827 26506 tablet_bootstrap.cc:492] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:34.118906 26506 ts_tablet_manager.cc:1367] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:34.119279 26506 raft_consensus.cc:348] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.119349 26506 raft_consensus.cc:374] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.119366 26506 raft_consensus.cc:729] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.119408 26504 ts_tablet_manager.cc:1398] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.119458 26506 consensus_queue.cc:260] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.119566 26504 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:34.119678 26506 ts_tablet_manager.cc:1398] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.119808 26506 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:34.120384 26504 tablet_bootstrap.cc:654] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.120579 26506 tablet_bootstrap.cc:654] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.120903 26502 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:34.121733 26502 tablet_bootstrap.cc:654] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.122133 26506 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:34.122215 26506 ts_tablet_manager.cc:1367] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:34.122599 26506 raft_consensus.cc:348] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.122704 26506 raft_consensus.cc:374] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.122712 26502 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:34.122737 26506 raft_consensus.cc:729] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.122787 26502 ts_tablet_manager.cc:1367] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:34.122845 26506 consensus_queue.cc:260] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.123133 26506 ts_tablet_manager.cc:1398] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.123133 26502 raft_consensus.cc:348] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.123205 26502 raft_consensus.cc:374] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.123224 26502 raft_consensus.cc:729] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.123313 26502 consensus_queue.cc:260] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.123556 26502 ts_tablet_manager.cc:1398] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:34.123733 26502 tablet_bootstrap.cc:492] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:34.124354 26503 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:34.124451 26503 ts_tablet_manager.cc:1367] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:34.124806 26502 tablet_bootstrap.cc:654] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.124919 26503 raft_consensus.cc:348] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.125012 26503 raft_consensus.cc:374] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.125048 26503 raft_consensus.cc:729] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.125178 26503 consensus_queue.cc:260] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.125530 26503 ts_tablet_manager.cc:1398] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.125768 26504 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:34.125836 26504 ts_tablet_manager.cc:1367] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:34.125911 26503 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:34.126428 26502 tablet_bootstrap.cc:492] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:34.126369 26504 raft_consensus.cc:348] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.126487 26504 raft_consensus.cc:374] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.126528 26504 raft_consensus.cc:729] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.126634 26502 ts_tablet_manager.cc:1367] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:34.126638 26504 consensus_queue.cc:260] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.126906 26504 ts_tablet_manager.cc:1398] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:34.127013 26503 tablet_bootstrap.cc:654] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.127074 26504 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:34.127511 26502 raft_consensus.cc:348] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.127664 26502 raft_consensus.cc:374] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.127739 26502 raft_consensus.cc:729] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.127936 26502 consensus_queue.cc:260] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.128252 26502 ts_tablet_manager.cc:1398] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:34.129251 26504 tablet_bootstrap.cc:654] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.130236 26504 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:34.130326 26504 ts_tablet_manager.cc:1367] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:34.130440 26503 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:34.130523 26503 ts_tablet_manager.cc:1367] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:34.130725 26504 raft_consensus.cc:348] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.130823 26504 raft_consensus.cc:374] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.130864 26504 raft_consensus.cc:729] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.130884 26503 raft_consensus.cc:348] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.130986 26503 raft_consensus.cc:374] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.131026 26503 raft_consensus.cc:729] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.130990 26504 consensus_queue.cc:260] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.131148 26503 consensus_queue.cc:260] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.131259 26504 ts_tablet_manager.cc:1398] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:34.131407 26503 ts_tablet_manager.cc:1398] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:34.211524 26494 raft_consensus.cc:482] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.211642 26494 raft_consensus.cc:504] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.212015 26494 leader_election.cc:288] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:34.212266 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3e57ee511e4d9886fcfed3851e4c38" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:34.212386 24897 raft_consensus.cc:2454] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:34.212361 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3e57ee511e4d9886fcfed3851e4c38" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:34.212479 24631 raft_consensus.cc:2454] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:34.212644 24966 leader_election.cc:302] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.212787 26494 raft_consensus.cc:2781] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.212842 26494 raft_consensus.cc:482] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.212879 26494 raft_consensus.cc:3037] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.213712 26494 raft_consensus.cc:504] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.214051 26494 leader_election.cc:288] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:34.214299 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3e57ee511e4d9886fcfed3851e4c38" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:34.214356 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa3e57ee511e4d9886fcfed3851e4c38" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:34.214408 24897 raft_consensus.cc:3037] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.214457 24631 raft_consensus.cc:3037] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.215315 24631 raft_consensus.cc:2454] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:34.215315 24897 raft_consensus.cc:2454] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:34.215600 24966 leader_election.cc:302] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.215768 26494 raft_consensus.cc:2781] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.215855 26494 raft_consensus.cc:686] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:34.215986 26494 consensus_queue.cc:237] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.217604 24480 catalog_manager.cc:5484] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:34.328174 26306 raft_consensus.cc:482] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.328279 26306 raft_consensus.cc:504] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.328590 26306 leader_election.cc:288] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:34.328907 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f999b09b79a4632936833f351c4b28b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:34.328946 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f999b09b79a4632936833f351c4b28b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:34.329030 24631 raft_consensus.cc:2454] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:34.329072 25034 raft_consensus.cc:2454] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:34.329300 24700 leader_election.cc:302] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:34.329460 26306 raft_consensus.cc:2781] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.329521 26306 raft_consensus.cc:482] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.329548 26306 raft_consensus.cc:3037] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.330392 26306 raft_consensus.cc:504] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.330698 26306 leader_election.cc:288] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:34.331017 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f999b09b79a4632936833f351c4b28b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:34.331038 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f999b09b79a4632936833f351c4b28b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:34.331115 24631 raft_consensus.cc:3037] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.331122 25034 raft_consensus.cc:3037] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.332024 25034 raft_consensus.cc:2454] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:34.332021 24631 raft_consensus.cc:2454] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:34.332291 24698 leader_election.cc:302] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:34.332439 26306 raft_consensus.cc:2781] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.332636 26306 raft_consensus.cc:686] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:34.332810 26306 consensus_queue.cc:237] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.334570 24480 catalog_manager.cc:5484] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:34.449024 26427 raft_consensus.cc:482] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.449128 26427 raft_consensus.cc:504] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.449429 26427 leader_election.cc:288] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:34.449754 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ede6fc65d044804848e6f61e8c238c0" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:34.449785 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ede6fc65d044804848e6f61e8c238c0" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:34.449898 24897 raft_consensus.cc:2454] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:34.449914 24765 raft_consensus.cc:2454] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:34.450229 24567 leader_election.cc:302] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.450399 26427 raft_consensus.cc:2781] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.450459 26427 raft_consensus.cc:482] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.450497 26427 raft_consensus.cc:3037] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.451259 26427 raft_consensus.cc:504] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.451572 26427 leader_election.cc:288] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:34.451843 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ede6fc65d044804848e6f61e8c238c0" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:34.451957 24897 raft_consensus.cc:3037] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.451930 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ede6fc65d044804848e6f61e8c238c0" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:34.452039 24765 raft_consensus.cc:3037] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.452890 24897 raft_consensus.cc:2454] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:34.453023 24765 raft_consensus.cc:2454] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:34.453158 24567 leader_election.cc:302] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.453289 26427 raft_consensus.cc:2781] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.453446 26427 raft_consensus.cc:686] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:34.453594 26427 consensus_queue.cc:237] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.455421 24480 catalog_manager.cc:5484] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:34.581871 26520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:34.582129 26520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:34.585264 26520 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 01:58:34.672459 24631 raft_consensus.cc:1261] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.672789 24897 raft_consensus.cc:1261] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.673180 26494 consensus_queue.cc:1035] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.673393 26297 consensus_queue.cc:1035] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.674185 24765 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.674594 26427 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.675484 24898 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.679484 25999 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.685619 25034 raft_consensus.cc:1261] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.685616 24632 raft_consensus.cc:1261] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:34.686827 26518 consensus_queue.cc:1035] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:34.687153 26518 consensus_queue.cc:1035] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.405 ms
time per row: 0.0622023 ms
I20240502 01:58:34.750169 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:34.755488 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 63312ec15b034e47bc2dd484b57d4648 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.755744 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63312ec15b034e47bc2dd484b57d4648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.755764 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 63312ec15b034e47bc2dd484b57d4648 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.756011 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63312ec15b034e47bc2dd484b57d4648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.756702 24875 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8db639839433d967583371351eacb (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.756925 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8db639839433d967583371351eacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.758503 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 645305c993a5486ebc1e6c8b81941128 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.758734 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645305c993a5486ebc1e6c8b81941128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.758862 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 645305c993a5486ebc1e6c8b81941128 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.759088 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645305c993a5486ebc1e6c8b81941128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.761250 26579 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:34.761966 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 645305c993a5486ebc1e6c8b81941128 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.762045 26579 tablet_bootstrap.cc:654] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.762183 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645305c993a5486ebc1e6c8b81941128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.762423 24745 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8db639839433d967583371351eacb (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.762595 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8db639839433d967583371351eacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.764501 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 63312ec15b034e47bc2dd484b57d4648 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.764585 26579 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:34.764590 25011 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8db639839433d967583371351eacb (DEFAULT_TABLE table=rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.764659 26579 ts_tablet_manager.cc:1367] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:34.764737 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63312ec15b034e47bc2dd484b57d4648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.764890 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8db639839433d967583371351eacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.764945 26578 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:34.765070 26579 raft_consensus.cc:348] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.765255 26579 raft_consensus.cc:374] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.765329 26579 raft_consensus.cc:729] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.765511 26579 consensus_queue.cc:260] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.765707 26578 tablet_bootstrap.cc:654] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.766990 26577 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:34.767311 26578 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:34.767342 26579 ts_tablet_manager.cc:1398] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:34.767383 26578 ts_tablet_manager.cc:1367] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:34.767799 26582 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:34.767796 26578 raft_consensus.cc:348] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.767887 26578 raft_consensus.cc:374] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.767930 26578 raft_consensus.cc:729] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.768033 26578 consensus_queue.cc:260] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.768267 26578 ts_tablet_manager.cc:1398] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.768520 26578 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:34.768594 26582 tablet_bootstrap.cc:654] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.768942 26577 tablet_bootstrap.cc:654] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.769346 26578 tablet_bootstrap.cc:654] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.769537 26582 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:34.769606 26582 ts_tablet_manager.cc:1367] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:34.770043 26582 raft_consensus.cc:348] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.770148 26582 raft_consensus.cc:374] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.770186 26582 raft_consensus.cc:729] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.770298 26582 consensus_queue.cc:260] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.770560 26582 ts_tablet_manager.cc:1398] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.770565 26578 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:34.770633 26578 ts_tablet_manager.cc:1367] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:34.770709 26582 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:34.771203 26578 raft_consensus.cc:348] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.771293 26578 raft_consensus.cc:374] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.771324 26578 raft_consensus.cc:729] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.771426 26578 consensus_queue.cc:260] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.771507 26582 tablet_bootstrap.cc:654] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.771659 26578 ts_tablet_manager.cc:1398] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.771795 26578 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:34.772845 26582 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:34.772927 26582 ts_tablet_manager.cc:1367] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:34.773262 26582 raft_consensus.cc:348] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.773336 26582 raft_consensus.cc:374] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.773363 26582 raft_consensus.cc:729] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.773479 26582 consensus_queue.cc:260] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.773715 26582 ts_tablet_manager.cc:1398] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:34.774169 26578 tablet_bootstrap.cc:654] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.774446 26577 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:34.774531 26577 ts_tablet_manager.cc:1367] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:34.774945 26577 raft_consensus.cc:348] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.775051 26577 raft_consensus.cc:374] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.775087 26577 raft_consensus.cc:729] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.775211 26577 consensus_queue.cc:260] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.775437 26578 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:34.775483 26577 ts_tablet_manager.cc:1398] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:34.775513 26578 ts_tablet_manager.cc:1367] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:34.775869 26578 raft_consensus.cc:348] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.775977 26578 raft_consensus.cc:374] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.776008 26578 raft_consensus.cc:729] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.776121 26578 consensus_queue.cc:260] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.776414 26578 ts_tablet_manager.cc:1398] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.776891 26577 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:34.777945 26516 raft_consensus.cc:482] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.778023 26516 raft_consensus.cc:504] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.778369 26516 leader_election.cc:288] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:34.778645 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8db639839433d967583371351eacb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:34.778743 25034 raft_consensus.cc:2454] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:34.778743 26577 tablet_bootstrap.cc:654] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.778739 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8db639839433d967583371351eacb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:34.779132 24698 leader_election.cc:302] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
W20240502 01:58:34.779281 24701 leader_election.cc:341] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:34.779603 26516 raft_consensus.cc:2781] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.779680 26516 raft_consensus.cc:482] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.779726 26516 raft_consensus.cc:3037] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.780121 26577 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:34.780174 26577 ts_tablet_manager.cc:1367] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:34.780450 26516 raft_consensus.cc:504] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.780472 26577 raft_consensus.cc:348] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.780532 26577 raft_consensus.cc:374] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.780550 26577 raft_consensus.cc:729] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.780635 26577 consensus_queue.cc:260] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.780737 26516 leader_election.cc:288] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:34.780920 26577 ts_tablet_manager.cc:1398] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.781054 26577 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:34.781261 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8db639839433d967583371351eacb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:34.781347 25034 raft_consensus.cc:3037] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.781312 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8db639839433d967583371351eacb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:34.781440 24896 raft_consensus.cc:3037] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.782081 26577 tablet_bootstrap.cc:654] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.782220 25034 raft_consensus.cc:2454] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:34.782244 24896 raft_consensus.cc:2454] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:34.782509 24698 leader_election.cc:302] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:34.782639 26516 raft_consensus.cc:2781] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.782711 26516 raft_consensus.cc:686] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:34.782837 26516 consensus_queue.cc:237] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:34.783761 26577 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:34.783843 26577 ts_tablet_manager.cc:1367] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:34.784226 26577 raft_consensus.cc:348] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.784323 26577 raft_consensus.cc:374] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.784361 26577 raft_consensus.cc:729] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.784469 26577 consensus_queue.cc:260] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.784574 24480 catalog_manager.cc:5484] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:34.784768 26577 ts_tablet_manager.cc:1398] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:34.902163 26306 raft_consensus.cc:482] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.902279 26306 raft_consensus.cc:504] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.902649 26306 leader_election.cc:288] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:34.902974 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63312ec15b034e47bc2dd484b57d4648" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:34.903060 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63312ec15b034e47bc2dd484b57d4648" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:34.903120 24896 raft_consensus.cc:2454] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:34.903167 25034 raft_consensus.cc:2454] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:34.903370 24701 leader_election.cc:302] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.903545 26306 raft_consensus.cc:2781] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.903610 26306 raft_consensus.cc:482] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.903646 26306 raft_consensus.cc:3037] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.904620 26306 raft_consensus.cc:504] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.904950 26306 leader_election.cc:288] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:34.905223 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63312ec15b034e47bc2dd484b57d4648" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:34.905236 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63312ec15b034e47bc2dd484b57d4648" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:34.905323 25034 raft_consensus.cc:3037] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.905329 24896 raft_consensus.cc:3037] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.906060 24896 raft_consensus.cc:2454] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:34.906222 25034 raft_consensus.cc:2454] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:34.906337 24701 leader_election.cc:302] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.906596 26306 raft_consensus.cc:2781] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.906677 26306 raft_consensus.cc:686] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:34.906795 26306 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:34.908746 24482 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:34.989638 26560 raft_consensus.cc:482] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.989751 26560 raft_consensus.cc:504] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.990170 26560 leader_election.cc:288] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:34.990490 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645305c993a5486ebc1e6c8b81941128" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:34.990511 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645305c993a5486ebc1e6c8b81941128" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:34.990612 24765 raft_consensus.cc:2454] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:34.990634 24896 raft_consensus.cc:2454] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:34.990881 24567 leader_election.cc:302] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:34.991048 26560 raft_consensus.cc:2781] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.991103 26560 raft_consensus.cc:482] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.991160 26560 raft_consensus.cc:3037] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.991971 26560 raft_consensus.cc:504] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.992252 26560 leader_election.cc:288] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:34.992534 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645305c993a5486ebc1e6c8b81941128" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:34.992589 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645305c993a5486ebc1e6c8b81941128" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:34.992681 24896 raft_consensus.cc:3037] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.992683 24765 raft_consensus.cc:3037] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.993573 24765 raft_consensus.cc:2454] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:34.993577 24896 raft_consensus.cc:2454] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:34.993875 24567 leader_election.cc:302] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:34.994057 26560 raft_consensus.cc:2781] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.994168 26560 raft_consensus.cc:686] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:34.994318 26560 consensus_queue.cc:237] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:34.996183 24482 catalog_manager.cc:5484] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:35.148736 26593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:35.148926 26593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:35.152412 26593 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 01:58:35.246546 24765 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.246546 24896 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.246613 25034 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.247020 26516 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.247171 26560 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.247627 25999 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.249616 24896 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.255537 26516 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.256891 24897 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.257297 25032 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.257689 26306 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.261026 26516 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.454 ms
time per row: 0.0607269 ms
I20240502 01:58:35.317072 24485 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:35.322093 25011 tablet_service.cc:1467] Processing CreateTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.322238 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.322214 24745 tablet_service.cc:1467] Processing CreateTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.322363 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2284e66dfe44567b2170e0fcff27bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.322474 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2284e66dfe44567b2170e0fcff27bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.322476 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a7e0e4e54f47849bb29d1e7ae029af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.324518 26651 tablet_bootstrap.cc:492] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:35.324486 24611 tablet_service.cc:1467] Processing CreateTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.324615 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.324745 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2284e66dfe44567b2170e0fcff27bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.324811 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a7e0e4e54f47849bb29d1e7ae029af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.325544 26651 tablet_bootstrap.cc:654] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.326926 26651 tablet_bootstrap.cc:492] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:35.327010 26651 ts_tablet_manager.cc:1367] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:35.327577 26652 tablet_bootstrap.cc:492] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:35.327605 24745 tablet_service.cc:1467] Processing CreateTablet for tablet ee784fe0525240078fc79d0af2756b8d (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.327792 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee784fe0525240078fc79d0af2756b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.327800 26651 raft_consensus.cc:348] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.328069 26651 raft_consensus.cc:374] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.328114 26651 raft_consensus.cc:729] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.328099 24610 tablet_service.cc:1467] Processing CreateTablet for tablet ee784fe0525240078fc79d0af2756b8d (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.328224 26651 consensus_queue.cc:260] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.328321 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee784fe0525240078fc79d0af2756b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.328629 26651 ts_tablet_manager.cc:1398] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:35.328825 26654 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:35.329303 26652 tablet_bootstrap.cc:654] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.329301 26651 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:35.330226 26651 tablet_bootstrap.cc:654] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.330314 26652 tablet_bootstrap.cc:492] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:35.330402 26652 ts_tablet_manager.cc:1367] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:35.330691 25012 tablet_service.cc:1467] Processing CreateTablet for tablet ee784fe0525240078fc79d0af2756b8d (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.330812 26652 raft_consensus.cc:348] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.330899 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee784fe0525240078fc79d0af2756b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.330900 26652 raft_consensus.cc:374] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.330940 26652 raft_consensus.cc:729] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.331044 26652 consensus_queue.cc:260] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.331312 26652 ts_tablet_manager.cc:1398] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.331456 26652 tablet_bootstrap.cc:492] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:35.331470 26654 tablet_bootstrap.cc:654] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.331521 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af (DEFAULT_TABLE table=rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.331727 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a7e0e4e54f47849bb29d1e7ae029af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.332419 26652 tablet_bootstrap.cc:654] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.332465 26654 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:35.332542 26654 ts_tablet_manager.cc:1367] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:35.332978 26654 raft_consensus.cc:348] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.333068 26654 raft_consensus.cc:374] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.333094 26654 raft_consensus.cc:729] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.333200 26654 consensus_queue.cc:260] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.333457 26654 ts_tablet_manager.cc:1398] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.334290 26652 tablet_bootstrap.cc:492] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:35.334378 26652 ts_tablet_manager.cc:1367] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:35.334592 26655 tablet_bootstrap.cc:492] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:35.334753 26652 raft_consensus.cc:348] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.334838 26652 raft_consensus.cc:374] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.334867 26652 raft_consensus.cc:729] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.334990 26652 consensus_queue.cc:260] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.335260 26652 ts_tablet_manager.cc:1398] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:35.336246 26655 tablet_bootstrap.cc:654] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.337167 26651 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:35.337245 26651 ts_tablet_manager.cc:1367] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:35.337563 26651 raft_consensus.cc:348] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.337630 26651 raft_consensus.cc:374] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.337653 26651 raft_consensus.cc:729] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.337749 26651 consensus_queue.cc:260] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.338135 26651 ts_tablet_manager.cc:1398] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.338286 26651 tablet_bootstrap.cc:492] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:35.339121 26651 tablet_bootstrap.cc:654] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.339959 26655 tablet_bootstrap.cc:492] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:35.340087 26655 ts_tablet_manager.cc:1367] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:35.340131 26651 tablet_bootstrap.cc:492] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:35.340194 26651 ts_tablet_manager.cc:1367] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:35.340543 26651 raft_consensus.cc:348] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.340524 26655 raft_consensus.cc:348] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.340631 26655 raft_consensus.cc:374] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.340629 26651 raft_consensus.cc:374] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.340659 26655 raft_consensus.cc:729] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.340659 26651 raft_consensus.cc:729] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.340768 26651 consensus_queue.cc:260] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.340819 26655 consensus_queue.cc:260] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.341069 26655 ts_tablet_manager.cc:1398] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.341068 26651 ts_tablet_manager.cc:1398] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.341292 26655 tablet_bootstrap.cc:492] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:35.342180 26655 tablet_bootstrap.cc:654] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.343461 26655 tablet_bootstrap.cc:492] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:35.343650 26655 ts_tablet_manager.cc:1367] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:35.343991 26655 raft_consensus.cc:348] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.344066 26655 raft_consensus.cc:374] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.344094 26655 raft_consensus.cc:729] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.344201 26655 consensus_queue.cc:260] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.344549 26655 ts_tablet_manager.cc:1398] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.344799 26655 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:35.345695 26655 tablet_bootstrap.cc:654] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.346765 26655 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:35.346839 26655 ts_tablet_manager.cc:1367] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:35.347198 26655 raft_consensus.cc:348] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.347288 26655 raft_consensus.cc:374] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.347317 26655 raft_consensus.cc:729] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.347437 26655 consensus_queue.cc:260] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.347651 26655 ts_tablet_manager.cc:1398] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:35.379709 26645 raft_consensus.cc:482] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:35.379789 26645 raft_consensus.cc:504] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.380107 26645 leader_election.cc:288] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:35.380367 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2284e66dfe44567b2170e0fcff27bd6" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:35.380427 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2284e66dfe44567b2170e0fcff27bd6" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:35.380507 25032 raft_consensus.cc:2454] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:35.380525 24632 raft_consensus.cc:2454] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:35.380790 24700 leader_election.cc:302] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:35.380960 26645 raft_consensus.cc:2781] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:35.381017 26645 raft_consensus.cc:482] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:35.381042 26645 raft_consensus.cc:3037] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.381851 26645 raft_consensus.cc:504] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.382160 26645 leader_election.cc:288] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:35.382440 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2284e66dfe44567b2170e0fcff27bd6" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:35.382437 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2284e66dfe44567b2170e0fcff27bd6" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:35.382540 24632 raft_consensus.cc:3037] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.382544 25032 raft_consensus.cc:3037] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.383200 25032 raft_consensus.cc:2454] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:35.383306 24632 raft_consensus.cc:2454] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:35.383489 24698 leader_election.cc:302] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:35.383623 26645 raft_consensus.cc:2781] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:35.383709 26645 raft_consensus.cc:686] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:35.383831 26645 consensus_queue.cc:237] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.385576 24478 catalog_manager.cc:5484] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:35.450067 26573 raft_consensus.cc:482] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:35.450179 26573 raft_consensus.cc:504] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.450513 26573 leader_election.cc:288] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:35.450791 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee784fe0525240078fc79d0af2756b8d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:35.450848 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee784fe0525240078fc79d0af2756b8d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:35.450920 24632 raft_consensus.cc:2454] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:35.450995 24765 raft_consensus.cc:2454] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:35.451212 24965 leader_election.cc:302] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:35.451449 26573 raft_consensus.cc:2781] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:35.451495 26573 raft_consensus.cc:482] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:35.451532 26573 raft_consensus.cc:3037] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.452379 26573 raft_consensus.cc:504] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.452713 26573 leader_election.cc:288] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:35.453019 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee784fe0525240078fc79d0af2756b8d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:35.453145 24632 raft_consensus.cc:3037] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.453181 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee784fe0525240078fc79d0af2756b8d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:35.453290 24765 raft_consensus.cc:3037] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.454155 24632 raft_consensus.cc:2454] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:35.454200 24765 raft_consensus.cc:2454] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:35.454420 24965 leader_election.cc:302] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:35.454682 26666 raft_consensus.cc:2781] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:35.454766 26666 raft_consensus.cc:686] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:35.454903 26666 consensus_queue.cc:237] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.456880 24478 catalog_manager.cc:5484] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:35.564754 26666 raft_consensus.cc:482] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:35.564864 26666 raft_consensus.cc:504] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.565232 26666 leader_election.cc:288] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:35.565469 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a7e0e4e54f47849bb29d1e7ae029af" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:35.565549 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a7e0e4e54f47849bb29d1e7ae029af" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:35.565593 24897 raft_consensus.cc:2454] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:35.565680 24632 raft_consensus.cc:2454] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:35.565833 24966 leader_election.cc:302] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:35.566022 26666 raft_consensus.cc:2781] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:35.566079 26666 raft_consensus.cc:482] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:35.566125 26666 raft_consensus.cc:3037] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.566957 26666 raft_consensus.cc:504] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.567256 26666 leader_election.cc:288] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:35.567569 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a7e0e4e54f47849bb29d1e7ae029af" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:35.567684 24632 raft_consensus.cc:3037] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.567664 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a7e0e4e54f47849bb29d1e7ae029af" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:35.567756 24897 raft_consensus.cc:3037] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.568609 24632 raft_consensus.cc:2454] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:35.568611 24897 raft_consensus.cc:2454] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:35.568925 24966 leader_election.cc:302] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:35.569077 26666 raft_consensus.cc:2781] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:35.569144 26666 raft_consensus.cc:686] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:35.569332 26666 consensus_queue.cc:237] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.573614 24478 catalog_manager.cc:5484] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:35.723577 26669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:35.723816 26669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:35.726799 26669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240502 01:58:35.819098 24632 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.819175 24897 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.819702 26573 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.819948 26666 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.822121 24631 raft_consensus.cc:1261] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.822633 26645 consensus_queue.cc:1035] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.823697 25032 raft_consensus.cc:1261] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.824535 26645 consensus_queue.cc:1035] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.825531 24765 raft_consensus.cc:1261] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.826552 26573 consensus_queue.cc:1035] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:35.828272 24630 raft_consensus.cc:1261] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:35.828732 26573 consensus_queue.cc:1035] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.973 ms
time per row: 0.0559867 ms
I20240502 01:58:35.877830 26702 mvcc.cc:205] Tried to move back new op lower bound from 7023063514412761088 to 7023063512613552128. Current Snapshot: MvccSnapshot[applied={T|T < 7023063514400604160}]
I20240502 01:58:35.884573 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:35.889743 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 0e47667fc8fa49e6b18ee3df06cda00a (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.889879 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 0e47667fc8fa49e6b18ee3df06cda00a (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.890069 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e47667fc8fa49e6b18ee3df06cda00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.890197 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e47667fc8fa49e6b18ee3df06cda00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.890812 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 0e47667fc8fa49e6b18ee3df06cda00a (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.890866 24875 tablet_service.cc:1467] Processing CreateTablet for tablet ce35d9953e294b4986ff9e93e12db6f6 (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.891072 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e47667fc8fa49e6b18ee3df06cda00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.891079 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce35d9953e294b4986ff9e93e12db6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.892484 26730 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:35.893074 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 91d0d9525b5f4384861dc50d0bf4967d (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.893281 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0d9525b5f4384861dc50d0bf4967d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.893441 26730 tablet_bootstrap.cc:654] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.893388 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 91d0d9525b5f4384861dc50d0bf4967d (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.893608 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0d9525b5f4384861dc50d0bf4967d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.894250 25012 tablet_service.cc:1467] Processing CreateTablet for tablet ce35d9953e294b4986ff9e93e12db6f6 (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.894497 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce35d9953e294b4986ff9e93e12db6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.894793 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 91d0d9525b5f4384861dc50d0bf4967d (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.894961 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0d9525b5f4384861dc50d0bf4967d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.896309 26734 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:35.896929 24611 tablet_service.cc:1467] Processing CreateTablet for tablet ce35d9953e294b4986ff9e93e12db6f6 (DEFAULT_TABLE table=rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:35.897146 26734 tablet_bootstrap.cc:654] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.897176 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce35d9953e294b4986ff9e93e12db6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:35.897477 26730 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:35.897554 26730 ts_tablet_manager.cc:1367] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:35.897944 26730 raft_consensus.cc:348] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.898053 26730 raft_consensus.cc:374] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.898082 26730 raft_consensus.cc:729] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.898196 26730 consensus_queue.cc:260] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.896903 26731 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:35.899832 26731 tablet_bootstrap.cc:654] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.900398 26730 ts_tablet_manager.cc:1398] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:35.900841 26731 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:35.900918 26731 ts_tablet_manager.cc:1367] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:35.901201 26730 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:35.901321 26731 raft_consensus.cc:348] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.901422 26731 raft_consensus.cc:374] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.901449 26731 raft_consensus.cc:729] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.901571 26731 consensus_queue.cc:260] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.901830 26731 ts_tablet_manager.cc:1398] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:35.902015 26731 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:35.901923 26730 tablet_bootstrap.cc:654] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.902819 26731 tablet_bootstrap.cc:654] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.902820 26732 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:35.903594 26732 tablet_bootstrap.cc:654] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.904476 26731 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:35.904565 26731 ts_tablet_manager.cc:1367] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:35.904734 26732 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:35.904820 26732 ts_tablet_manager.cc:1367] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:35.904974 26731 raft_consensus.cc:348] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.905082 26731 raft_consensus.cc:374] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.905119 26731 raft_consensus.cc:729] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.905228 26731 consensus_queue.cc:260] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.905488 26731 ts_tablet_manager.cc:1398] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:35.906138 26732 raft_consensus.cc:348] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.906283 26732 raft_consensus.cc:374] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.906359 26734 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:35.906354 26732 raft_consensus.cc:729] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.906486 26734 ts_tablet_manager.cc:1367] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:35.906477 26732 consensus_queue.cc:260] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.906847 26732 ts_tablet_manager.cc:1398] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:35.906960 26734 raft_consensus.cc:348] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.907117 26734 raft_consensus.cc:374] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.907147 26732 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:35.907182 26734 raft_consensus.cc:729] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.907382 26734 consensus_queue.cc:260] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.907719 26734 ts_tablet_manager.cc:1398] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.908257 26732 tablet_bootstrap.cc:654] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.908485 26734 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:35.909544 26734 tablet_bootstrap.cc:654] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.910713 26730 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:35.910792 26730 ts_tablet_manager.cc:1367] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 01:58:35.911214 26730 raft_consensus.cc:348] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.911311 26730 raft_consensus.cc:374] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.911335 26730 raft_consensus.cc:729] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.911432 26730 consensus_queue.cc:260] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:35.911652 26730 ts_tablet_manager.cc:1398] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.911779 26730 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:35.912685 26730 tablet_bootstrap.cc:654] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:35.914429 26730 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:35.914512 26730 ts_tablet_manager.cc:1367] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:35.914907 26730 raft_consensus.cc:348] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.915006 26730 raft_consensus.cc:374] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.915040 26730 raft_consensus.cc:729] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.915139 26730 consensus_queue.cc:260] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.915364 26730 ts_tablet_manager.cc:1398] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:35.915823 26734 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:35.915903 26734 ts_tablet_manager.cc:1367] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:35.916288 26734 raft_consensus.cc:348] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.916380 26734 raft_consensus.cc:374] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.916405 26734 raft_consensus.cc:729] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.916520 26734 consensus_queue.cc:260] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.916832 26734 ts_tablet_manager.cc:1398] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:35.922595 26732 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:35.922688 26732 ts_tablet_manager.cc:1367] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240502 01:58:35.923252 26732 raft_consensus.cc:348] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.923345 26732 raft_consensus.cc:374] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:35.923374 26732 raft_consensus.cc:729] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:35.923496 26732 consensus_queue.cc:260] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:35.923785 26732 ts_tablet_manager.cc:1398] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:35.976302 26550 raft_consensus.cc:482] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:35.976413 26550 raft_consensus.cc:504] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.976737 26550 leader_election.cc:288] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:35.977010 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e47667fc8fa49e6b18ee3df06cda00a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:35.977056 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e47667fc8fa49e6b18ee3df06cda00a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:35.977165 24765 raft_consensus.cc:2454] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:35.977165 24632 raft_consensus.cc:2454] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:35.977398 24832 leader_election.cc:302] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:35.977555 26550 raft_consensus.cc:2781] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:35.977627 26550 raft_consensus.cc:482] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:35.977655 26550 raft_consensus.cc:3037] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.978547 26550 raft_consensus.cc:504] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.978878 26550 leader_election.cc:288] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:35.979148 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e47667fc8fa49e6b18ee3df06cda00a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:35.979282 24632 raft_consensus.cc:3037] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.979429 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e47667fc8fa49e6b18ee3df06cda00a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:35.979540 24765 raft_consensus.cc:3037] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:35.980077 24632 raft_consensus.cc:2454] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:35.980337 24832 leader_election.cc:302] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:35.980362 24765 raft_consensus.cc:2454] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:35.980489 26550 raft_consensus.cc:2781] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:35.980706 26550 raft_consensus.cc:686] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:35.980839 26550 consensus_queue.cc:237] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:35.982793 24479 catalog_manager.cc:5484] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:36.061738 26710 raft_consensus.cc:482] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:36.061861 26710 raft_consensus.cc:504] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.062340 26710 leader_election.cc:288] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:36.062649 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce35d9953e294b4986ff9e93e12db6f6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:36.062804 24897 raft_consensus.cc:2454] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:36.062808 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce35d9953e294b4986ff9e93e12db6f6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:36.062973 24632 raft_consensus.cc:2454] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:36.063100 24966 leader_election.cc:302] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:36.063257 26710 raft_consensus.cc:2781] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:36.063310 26710 raft_consensus.cc:482] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:36.063335 26710 raft_consensus.cc:3037] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.064174 26710 raft_consensus.cc:504] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.064499 26710 leader_election.cc:288] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:36.064854 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce35d9953e294b4986ff9e93e12db6f6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:36.064846 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce35d9953e294b4986ff9e93e12db6f6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:36.064958 24632 raft_consensus.cc:3037] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.064966 24897 raft_consensus.cc:3037] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.065683 24632 raft_consensus.cc:2454] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:36.065794 24897 raft_consensus.cc:2454] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:36.066066 24965 leader_election.cc:302] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:36.066249 26710 raft_consensus.cc:2781] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:36.066344 26710 raft_consensus.cc:686] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:36.066490 26710 consensus_queue.cc:237] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.068419 24479 catalog_manager.cc:5484] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:36.121457 26550 raft_consensus.cc:482] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:36.121584 26550 raft_consensus.cc:504] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:36.121973 26550 leader_election.cc:288] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:36.122296 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:36.122311 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:36.122423 25032 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:36.122433 24765 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:36.122686 24830 leader_election.cc:302] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:36.122855 26550 raft_consensus.cc:2781] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:36.122915 26550 raft_consensus.cc:482] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:36.122949 26550 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.123800 26550 raft_consensus.cc:504] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:36.124097 26550 leader_election.cc:288] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:36.124414 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:36.124437 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:36.124538 25032 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.124562 24765 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.125460 25032 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:36.125471 24765 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:36.125727 24833 leader_election.cc:302] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:36.125888 26550 raft_consensus.cc:2781] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:36.125983 26550 raft_consensus.cc:686] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:36.126148 26550 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:36.128119 24479 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:36.292593 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:36.292790 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:36.295838 26748 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 01:58:36.374392 24632 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.374943 24765 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.375006 26710 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.375569 26743 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.375877 24630 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.376279 26743 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.379590 25032 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.380257 26743 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.380635 24897 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.381587 26710 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.388401 24764 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.390293 26550 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.397596 26783 mvcc.cc:205] Tried to move back new op lower bound from 7023063516663021568 to 7023063515409473536. Current Snapshot: MvccSnapshot[applied={T|T < 7023063516663021568}]
I20240502 01:58:36.418623 26781 mvcc.cc:205] Tried to move back new op lower bound from 7023063516666314752 to 7023063515058626560. Current Snapshot: MvccSnapshot[applied={T|T < 7023063516666314752}]
INSERT report
rows total: 2000
time total: 116.526 ms
time per row: 0.058263 ms
I20240502 01:58:36.453470 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:36.458954 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 1a383190824e4611a9e279b05bfe4e2a (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.459138 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 1a383190824e4611a9e279b05bfe4e2a (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.459250 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a383190824e4611a9e279b05bfe4e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.459412 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a383190824e4611a9e279b05bfe4e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.459868 25011 tablet_service.cc:1467] Processing CreateTablet for tablet a094659d539e4c4f9861e356168c5894 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.460109 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a094659d539e4c4f9861e356168c5894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.461469 26811 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:36.461642 24745 tablet_service.cc:1467] Processing CreateTablet for tablet a094659d539e4c4f9861e356168c5894 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.461658 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 1a383190824e4611a9e279b05bfe4e2a (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.461953 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a094659d539e4c4f9861e356168c5894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.462098 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a383190824e4611a9e279b05bfe4e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.462370 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.462582 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d360cfdd64846b6cd37c9b859d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.462594 26811 tablet_bootstrap.cc:654] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.463280 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.463485 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d360cfdd64846b6cd37c9b859d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.464881 24611 tablet_service.cc:1467] Processing CreateTablet for tablet a094659d539e4c4f9861e356168c5894 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.464941 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:36.465095 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a094659d539e4c4f9861e356168c5894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.465162 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d360cfdd64846b6cd37c9b859d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:36.465317 26812 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:36.466188 26812 tablet_bootstrap.cc:654] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.466849 26813 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:36.467051 26810 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:36.467180 26812 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:36.467262 26812 ts_tablet_manager.cc:1367] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:36.467612 26813 tablet_bootstrap.cc:654] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.467676 26812 raft_consensus.cc:348] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.467789 26812 raft_consensus.cc:374] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.467804 26810 tablet_bootstrap.cc:654] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.467859 26812 raft_consensus.cc:729] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.468003 26812 consensus_queue.cc:260] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.468290 26812 ts_tablet_manager.cc:1398] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.468782 26810 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:36.468838 26813 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:36.468864 26810 ts_tablet_manager.cc:1367] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:36.468879 26812 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:36.468900 26813 ts_tablet_manager.cc:1367] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:36.469267 26810 raft_consensus.cc:348] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.469359 26810 raft_consensus.cc:374] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.469327 26813 raft_consensus.cc:348] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.469388 26810 raft_consensus.cc:729] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.469408 26813 raft_consensus.cc:374] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.469436 26813 raft_consensus.cc:729] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.469549 26812 tablet_bootstrap.cc:654] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.469511 26810 consensus_queue.cc:260] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.469548 26813 consensus_queue.cc:260] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.469775 26810 ts_tablet_manager.cc:1398] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.469852 26813 ts_tablet_manager.cc:1398] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.469918 26810 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:36.470002 26813 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:36.470675 26812 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:36.470726 26812 ts_tablet_manager.cc:1367] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:36.470739 26813 tablet_bootstrap.cc:654] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.471040 26812 raft_consensus.cc:348] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.471114 26812 raft_consensus.cc:374] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.471159 26812 raft_consensus.cc:729] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.471254 26812 consensus_queue.cc:260] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.471287 26811 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:36.471367 26811 ts_tablet_manager.cc:1367] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:36.471520 26812 ts_tablet_manager.cc:1398] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.471824 26813 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:36.471765 26811 raft_consensus.cc:348] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.471866 26811 raft_consensus.cc:374] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.471889 26813 ts_tablet_manager.cc:1367] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:36.471896 26811 raft_consensus.cc:729] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.472020 26811 consensus_queue.cc:260] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.472280 26811 ts_tablet_manager.cc:1398] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:36.472273 26813 raft_consensus.cc:348] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.472362 26813 raft_consensus.cc:374] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.472390 26813 raft_consensus.cc:729] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.472512 26811 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:36.472497 26813 consensus_queue.cc:260] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.472743 26813 ts_tablet_manager.cc:1398] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.473346 26811 tablet_bootstrap.cc:654] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.473832 26810 tablet_bootstrap.cc:654] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.474570 26811 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:36.474638 26811 ts_tablet_manager.cc:1367] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:36.475021 26811 raft_consensus.cc:348] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.475106 26811 raft_consensus.cc:374] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.475132 26811 raft_consensus.cc:729] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.475241 26811 consensus_queue.cc:260] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.475577 26811 ts_tablet_manager.cc:1398] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.475744 26811 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:36.476058 26810 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:36.476127 26810 ts_tablet_manager.cc:1367] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:58:36.476531 26811 tablet_bootstrap.cc:654] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:36.476501 26810 raft_consensus.cc:348] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.476588 26810 raft_consensus.cc:374] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.476615 26810 raft_consensus.cc:729] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.476729 26810 consensus_queue.cc:260] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.476979 26810 ts_tablet_manager.cc:1398] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:36.478755 26811 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:36.479516 26811 ts_tablet_manager.cc:1367] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:36.479983 26811 raft_consensus.cc:348] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.480110 26811 raft_consensus.cc:374] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:36.480154 26811 raft_consensus.cc:729] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:36.480288 26811 consensus_queue.cc:260] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.480607 26811 ts_tablet_manager.cc:1398] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:36.499387 26645 raft_consensus.cc:482] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:36.499519 26645 raft_consensus.cc:504] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.499914 26645 leader_election.cc:288] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:36.500280 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a383190824e4611a9e279b05bfe4e2a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:36.500327 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a383190824e4611a9e279b05bfe4e2a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:36.500418 24897 raft_consensus.cc:2454] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:36.500492 24630 raft_consensus.cc:2454] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:36.500721 24701 leader_election.cc:302] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:36.500946 26645 raft_consensus.cc:2781] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:36.501020 26645 raft_consensus.cc:482] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:36.501060 26645 raft_consensus.cc:3037] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.502117 26645 raft_consensus.cc:504] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.502467 26645 leader_election.cc:288] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:36.502777 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a383190824e4611a9e279b05bfe4e2a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:36.502801 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a383190824e4611a9e279b05bfe4e2a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:36.502895 24630 raft_consensus.cc:3037] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.502898 24897 raft_consensus.cc:3037] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.503643 24897 raft_consensus.cc:2454] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:36.503765 24630 raft_consensus.cc:2454] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:36.503962 24701 leader_election.cc:302] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:36.504135 26645 raft_consensus.cc:2781] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:36.504225 26645 raft_consensus.cc:686] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:36.504365 26645 consensus_queue.cc:237] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:36.506405 24485 catalog_manager.cc:5484] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:36.525311 26632 raft_consensus.cc:482] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:36.525426 26632 raft_consensus.cc:504] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.525846 26632 leader_election.cc:288] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:36.526166 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d360cfdd64846b6cd37c9b859d7a8" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:36.526304 24898 raft_consensus.cc:2454] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:36.526355 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d360cfdd64846b6cd37c9b859d7a8" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:36.526525 25032 raft_consensus.cc:2454] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:36.526597 24567 leader_election.cc:302] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:36.526760 26632 raft_consensus.cc:2781] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:36.526820 26632 raft_consensus.cc:482] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:36.526865 26632 raft_consensus.cc:3037] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.527657 26632 raft_consensus.cc:504] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.527969 26632 leader_election.cc:288] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:36.528234 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d360cfdd64846b6cd37c9b859d7a8" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:36.528312 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752d360cfdd64846b6cd37c9b859d7a8" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:36.528379 24898 raft_consensus.cc:3037] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.528425 25032 raft_consensus.cc:3037] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.529284 24898 raft_consensus.cc:2454] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:36.529289 25032 raft_consensus.cc:2454] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:36.529531 24564 leader_election.cc:302] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:36.529691 26632 raft_consensus.cc:2781] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:36.529774 26632 raft_consensus.cc:686] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:36.529969 26632 consensus_queue.cc:237] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:36.531771 24485 catalog_manager.cc:5484] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:36.742870 26632 raft_consensus.cc:482] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:36.742985 26632 raft_consensus.cc:504] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.743325 26632 leader_election.cc:288] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:36.743615 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a094659d539e4c4f9861e356168c5894" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:36.743619 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a094659d539e4c4f9861e356168c5894" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:36.743731 24765 raft_consensus.cc:2454] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:36.743739 25032 raft_consensus.cc:2454] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:36.743986 24567 leader_election.cc:302] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:36.744148 26632 raft_consensus.cc:2781] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:36.744211 26632 raft_consensus.cc:482] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:36.744251 26632 raft_consensus.cc:3037] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.745178 26632 raft_consensus.cc:504] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.745491 26632 leader_election.cc:288] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:36.745767 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a094659d539e4c4f9861e356168c5894" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:36.745872 25032 raft_consensus.cc:3037] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.745908 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a094659d539e4c4f9861e356168c5894" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:36.746057 24765 raft_consensus.cc:3037] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:36.746824 25032 raft_consensus.cc:2454] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:36.746901 24765 raft_consensus.cc:2454] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:36.747150 24564 leader_election.cc:302] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:36.747329 26632 raft_consensus.cc:2781] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:36.747419 26632 raft_consensus.cc:686] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:36.747555 26632 consensus_queue.cc:237] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:36.749480 24485 catalog_manager.cc:5484] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:36.863920 26826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:36.864112 26826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:36.867113 26826 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 01:58:36.945152 26645 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.949954 26516 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.951509 25032 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.951678 24765 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.951885 26745 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.953570 26745 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.959350 25034 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.959446 24897 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:36.959811 26745 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:36.960386 26745 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.333 ms
time per row: 0.0536663 ms
I20240502 01:58:37.020495 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:37.026135 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0cc9aba3804968a6dce97682f6b38e (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.026242 24745 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0cc9aba3804968a6dce97682f6b38e (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.026443 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0cc9aba3804968a6dce97682f6b38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.026531 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0cc9aba3804968a6dce97682f6b38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.026767 24743 tablet_service.cc:1467] Processing CreateTablet for tablet d136e006dd1048739156526af42fddd1 (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.026813 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 93695e2165e142f79293ac14e4f4fb84 (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.026957 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d136e006dd1048739156526af42fddd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.027118 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93695e2165e142f79293ac14e4f4fb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.027702 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0cc9aba3804968a6dce97682f6b38e (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.027948 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0cc9aba3804968a6dce97682f6b38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.029132 25012 tablet_service.cc:1467] Processing CreateTablet for tablet d136e006dd1048739156526af42fddd1 (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.029279 26889 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:37.029404 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d136e006dd1048739156526af42fddd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.030128 26892 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:37.030249 26889 tablet_bootstrap.cc:654] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.030194 24875 tablet_service.cc:1467] Processing CreateTablet for tablet d136e006dd1048739156526af42fddd1 (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.030406 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d136e006dd1048739156526af42fddd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.030968 26892 tablet_bootstrap.cc:654] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.032505 26892 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:37.032625 26892 ts_tablet_manager.cc:1367] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:37.032874 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 93695e2165e142f79293ac14e4f4fb84 (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.033077 26892 raft_consensus.cc:348] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.033128 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93695e2165e142f79293ac14e4f4fb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.033177 26892 raft_consensus.cc:374] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.033205 26892 raft_consensus.cc:729] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.033314 26892 consensus_queue.cc:260] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.033576 26892 ts_tablet_manager.cc:1398] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.033708 26892 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:37.034649 26892 tablet_bootstrap.cc:654] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.035601 26892 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:37.035667 26892 ts_tablet_manager.cc:1367] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:37.035993 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 93695e2165e142f79293ac14e4f4fb84 (DEFAULT_TABLE table=rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.036065 26892 raft_consensus.cc:348] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.036162 26892 raft_consensus.cc:374] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.036190 26892 raft_consensus.cc:729] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.036195 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93695e2165e142f79293ac14e4f4fb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.036296 26892 consensus_queue.cc:260] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.036561 26892 ts_tablet_manager.cc:1398] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.036696 26897 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:37.037365 26897 tablet_bootstrap.cc:654] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.037634 26892 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:37.038281 26890 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:37.038532 26897 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:37.038619 26897 ts_tablet_manager.cc:1367] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:37.038637 26892 tablet_bootstrap.cc:654] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.039043 26897 raft_consensus.cc:348] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.039117 26890 tablet_bootstrap.cc:654] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.039139 26897 raft_consensus.cc:374] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.039161 26897 raft_consensus.cc:729] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.039258 26897 consensus_queue.cc:260] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.039528 26897 ts_tablet_manager.cc:1398] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.040549 26889 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:37.040639 26889 ts_tablet_manager.cc:1367] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:58:37.040421 26890 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:37.040989 26890 ts_tablet_manager.cc:1367] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:37.041092 26889 raft_consensus.cc:348] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.041194 26889 raft_consensus.cc:374] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.041225 26889 raft_consensus.cc:729] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.041344 26889 consensus_queue.cc:260] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.041404 26890 raft_consensus.cc:348] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.041487 26890 raft_consensus.cc:374] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.041515 26890 raft_consensus.cc:729] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.041604 26889 ts_tablet_manager.cc:1398] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:37.041621 26890 consensus_queue.cc:260] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.041880 26890 ts_tablet_manager.cc:1398] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.041868 26889 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:37.042044 26890 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:37.042793 26889 tablet_bootstrap.cc:654] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.043900 26890 tablet_bootstrap.cc:654] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.044582 26892 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:37.044668 26892 ts_tablet_manager.cc:1367] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:37.045096 26892 raft_consensus.cc:348] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.045212 26892 raft_consensus.cc:374] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.045269 26892 raft_consensus.cc:729] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.045405 26892 consensus_queue.cc:260] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.045822 26890 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:37.045989 26890 ts_tablet_manager.cc:1367] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:37.045997 26892 ts_tablet_manager.cc:1398] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:37.046439 26890 raft_consensus.cc:348] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.046546 26890 raft_consensus.cc:374] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.046581 26890 raft_consensus.cc:729] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.046730 26890 consensus_queue.cc:260] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.046840 26889 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:37.046940 26889 ts_tablet_manager.cc:1367] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:37.047081 26890 ts_tablet_manager.cc:1398] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:37.047381 26889 raft_consensus.cc:348] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.047485 26889 raft_consensus.cc:374] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.047518 26889 raft_consensus.cc:729] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.047639 26889 consensus_queue.cc:260] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.047901 26889 ts_tablet_manager.cc:1398] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.048164 26889 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:37.049155 26889 tablet_bootstrap.cc:654] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.050413 26889 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:37.050493 26889 ts_tablet_manager.cc:1367] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:37.050626 26794 raft_consensus.cc:482] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:37.050724 26794 raft_consensus.cc:504] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.050873 26889 raft_consensus.cc:348] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.051004 26889 raft_consensus.cc:374] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.051060 26889 raft_consensus.cc:729] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.051167 26794 leader_election.cc:288] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:37.051277 26889 consensus_queue.cc:260] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.051578 26889 ts_tablet_manager.cc:1398] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:37.051651 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:37.051775 25034 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:37.052167 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:37.052268 24765 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:37.052562 24833 leader_election.cc:302] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.052925 26794 raft_consensus.cc:2781] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:37.052987 26794 raft_consensus.cc:482] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:37.053035 26794 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.053968 26794 raft_consensus.cc:504] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.054325 26794 leader_election.cc:288] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:37.054659 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:37.054755 25034 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.054867 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:37.054953 24765 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.055560 25034 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:37.055622 24765 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:37.055833 24830 leader_election.cc:302] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.055997 26794 raft_consensus.cc:2781] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:37.056123 26794 raft_consensus.cc:686] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:37.056272 26794 consensus_queue.cc:237] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.058180 24484 catalog_manager.cc:5484] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:37.162564 26793 raft_consensus.cc:482] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:37.162667 26793 raft_consensus.cc:504] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.163058 26793 leader_election.cc:288] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:37.163336 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0cc9aba3804968a6dce97682f6b38e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:37.163399 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0cc9aba3804968a6dce97682f6b38e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:37.163447 25034 raft_consensus.cc:2454] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:37.163543 24765 raft_consensus.cc:2454] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:37.163709 24830 leader_election.cc:302] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.163857 26793 raft_consensus.cc:2781] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:37.163916 26793 raft_consensus.cc:482] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:37.163964 26793 raft_consensus.cc:3037] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.164734 26793 raft_consensus.cc:504] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.165028 26793 leader_election.cc:288] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:37.165318 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0cc9aba3804968a6dce97682f6b38e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:37.165331 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0cc9aba3804968a6dce97682f6b38e" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:37.165411 25034 raft_consensus.cc:3037] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.165443 24765 raft_consensus.cc:3037] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.166158 25034 raft_consensus.cc:2454] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:37.166230 24765 raft_consensus.cc:2454] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:37.166420 24830 leader_election.cc:302] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.166569 26793 raft_consensus.cc:2781] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:37.166651 26793 raft_consensus.cc:686] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:37.166779 26793 consensus_queue.cc:237] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.168502 24484 catalog_manager.cc:5484] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:37.351816 26793 raft_consensus.cc:482] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:37.351933 26793 raft_consensus.cc:504] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.352283 26793 leader_election.cc:288] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:37.352629 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:37.352630 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:37.352771 24765 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:37.352771 24630 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:37.353082 24832 leader_election.cc:302] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.353320 26793 raft_consensus.cc:2781] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:37.353377 26793 raft_consensus.cc:482] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:37.353413 26793 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.354249 26793 raft_consensus.cc:504] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.354529 26793 leader_election.cc:288] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:37.354851 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:37.354851 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:37.354980 24765 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.354980 24630 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.355870 24630 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:37.355872 24765 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:37.356153 24832 leader_election.cc:302] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.356302 26793 raft_consensus.cc:2781] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:37.356392 26793 raft_consensus.cc:686] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:37.356521 26793 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:37.358597 24484 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:37.503692 26793 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:37.506764 26907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:37.507002 26907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:58:37.507339 26793 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:37.510639 26907 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 01:58:37.584987 25034 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:37.585172 24765 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:37.585500 26793 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:37.585796 26793 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:37.594858 24763 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:37.595284 26794 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:37.596141 24630 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:37.597281 26794 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:37.605366 26939 mvcc.cc:205] Tried to move back new op lower bound from 7023063521624584192 to 7023063519915171840. Current Snapshot: MvccSnapshot[applied={T|T < 7023063521624584192}]
INSERT report
rows total: 2000
time total: 109.89 ms
time per row: 0.054945 ms
I20240502 01:58:37.657795 24478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:37.663419 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 693e0959e61c44c59f797e4060187cef (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.663504 24877 tablet_service.cc:1467] Processing CreateTablet for tablet e148806c0b9143c18ee7938479b481b3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.663739 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e148806c0b9143c18ee7938479b481b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.663740 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693e0959e61c44c59f797e4060187cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.663913 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 693e0959e61c44c59f797e4060187cef (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.664149 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693e0959e61c44c59f797e4060187cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.666152 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.666152 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 693e0959e61c44c59f797e4060187cef (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.666257 26963 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:37.666385 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693e0959e61c44c59f797e4060187cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.666162 24610 tablet_service.cc:1467] Processing CreateTablet for tablet e148806c0b9143c18ee7938479b481b3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.667028 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e148806c0b9143c18ee7938479b481b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.667109 26963 tablet_bootstrap.cc:654] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.667157 26965 tablet_bootstrap.cc:492] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:37.667856 26965 tablet_bootstrap.cc:654] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.668390 26964 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:37.668743 26968 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:37.669169 26964 tablet_bootstrap.cc:654] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.669266 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751feb4b2419463c81f3571f6eb0fa42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.669551 26965 tablet_bootstrap.cc:492] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:37.669608 26965 ts_tablet_manager.cc:1367] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:37.669754 26968 tablet_bootstrap.cc:654] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.669914 26965 raft_consensus.cc:348] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.670060 26965 raft_consensus.cc:374] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.670109 26965 raft_consensus.cc:729] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.670246 26965 consensus_queue.cc:260] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.670562 26965 ts_tablet_manager.cc:1398] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:37.670775 25012 tablet_service.cc:1467] Processing CreateTablet for tablet e148806c0b9143c18ee7938479b481b3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.670868 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.670981 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e148806c0b9143c18ee7938479b481b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.671088 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751feb4b2419463c81f3571f6eb0fa42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.672480 26964 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:37.672616 26964 ts_tablet_manager.cc:1367] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:37.672924 26968 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:37.673053 26963 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:37.673141 26963 ts_tablet_manager.cc:1367] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:58:37.673230 26964 raft_consensus.cc:348] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.673342 26968 ts_tablet_manager.cc:1367] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:37.673403 26964 raft_consensus.cc:374] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.673492 26964 raft_consensus.cc:729] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.673511 26963 raft_consensus.cc:348] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.673715 26964 consensus_queue.cc:260] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.673796 26968 raft_consensus.cc:348] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.673874 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 (DEFAULT_TABLE table=rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:37.674177 26964 ts_tablet_manager.cc:1398] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:37.673921 26968 raft_consensus.cc:374] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.674650 26968 raft_consensus.cc:729] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.674717 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751feb4b2419463c81f3571f6eb0fa42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:37.674783 26968 consensus_queue.cc:260] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.675141 26968 ts_tablet_manager.cc:1398] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:37.675444 26968 tablet_bootstrap.cc:492] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:37.676754 26963 raft_consensus.cc:374] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.676821 26963 raft_consensus.cc:729] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.676939 26963 consensus_queue.cc:260] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.677227 26968 tablet_bootstrap.cc:654] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.677767 26963 ts_tablet_manager.cc:1398] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:37.677964 26963 tablet_bootstrap.cc:492] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:37.678443 26968 tablet_bootstrap.cc:492] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:37.678519 26968 ts_tablet_manager.cc:1367] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:37.678886 26968 raft_consensus.cc:348] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.678982 26968 raft_consensus.cc:374] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.679013 26968 raft_consensus.cc:729] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.679117 26968 consensus_queue.cc:260] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.679375 26968 ts_tablet_manager.cc:1398] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.679548 26968 tablet_bootstrap.cc:492] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:37.680195 26963 tablet_bootstrap.cc:654] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.680315 26968 tablet_bootstrap.cc:654] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.678794 26964 tablet_bootstrap.cc:492] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:37.681366 26968 tablet_bootstrap.cc:492] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:37.681411 26963 tablet_bootstrap.cc:492] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:37.681437 26968 ts_tablet_manager.cc:1367] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:37.681478 26963 ts_tablet_manager.cc:1367] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:37.681710 26964 tablet_bootstrap.cc:654] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.681793 26968 raft_consensus.cc:348] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.681820 26963 raft_consensus.cc:348] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.681871 26968 raft_consensus.cc:374] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.681901 26968 raft_consensus.cc:729] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.682022 26968 consensus_queue.cc:260] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.682297 26968 ts_tablet_manager.cc:1398] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:37.681907 26963 raft_consensus.cc:374] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.682682 26963 raft_consensus.cc:729] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.682806 26963 consensus_queue.cc:260] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.683022 26964 tablet_bootstrap.cc:492] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:37.683100 26964 ts_tablet_manager.cc:1367] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:37.683094 26963 ts_tablet_manager.cc:1398] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:37.683264 26963 tablet_bootstrap.cc:492] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:37.683493 26964 raft_consensus.cc:348] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.683583 26964 raft_consensus.cc:374] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.683612 26964 raft_consensus.cc:729] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.683705 26964 consensus_queue.cc:260] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.683964 26964 ts_tablet_manager.cc:1398] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:37.683987 26963 tablet_bootstrap.cc:654] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:37.684921 26963 tablet_bootstrap.cc:492] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:37.685007 26963 ts_tablet_manager.cc:1367] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:37.685321 26963 raft_consensus.cc:348] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.685400 26963 raft_consensus.cc:374] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:37.685427 26963 raft_consensus.cc:729] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:37.685544 26963 consensus_queue.cc:260] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.685770 26963 ts_tablet_manager.cc:1398] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:37.748664 26666 raft_consensus.cc:482] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:37.748797 26666 raft_consensus.cc:504] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.749186 26666 leader_election.cc:288] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:37.749492 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693e0959e61c44c59f797e4060187cef" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:37.749508 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693e0959e61c44c59f797e4060187cef" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:37.749619 24630 raft_consensus.cc:2454] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:37.749646 24765 raft_consensus.cc:2454] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:37.749964 24965 leader_election.cc:302] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:37.750147 26666 raft_consensus.cc:2781] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:37.750200 26666 raft_consensus.cc:482] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:37.750236 26666 raft_consensus.cc:3037] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.751078 26666 raft_consensus.cc:504] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.751435 26666 leader_election.cc:288] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:37.751776 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693e0959e61c44c59f797e4060187cef" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:37.751755 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693e0959e61c44c59f797e4060187cef" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:37.751883 24765 raft_consensus.cc:3037] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.751888 24630 raft_consensus.cc:3037] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.752774 24765 raft_consensus.cc:2454] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:37.752789 24630 raft_consensus.cc:2454] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:37.753088 24965 leader_election.cc:302] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:37.753162 26666 raft_consensus.cc:482] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:37.753221 26666 raft_consensus.cc:504] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.753533 26666 leader_election.cc:288] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:37.753628 26666 raft_consensus.cc:2781] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:37.753981 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e148806c0b9143c18ee7938479b481b3" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:37.754105 24897 raft_consensus.cc:2454] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:37.754119 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e148806c0b9143c18ee7938479b481b3" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:37.754218 24630 raft_consensus.cc:2454] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:37.753683 26666 raft_consensus.cc:686] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:37.754408 24966 leader_election.cc:302] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.754721 26891 raft_consensus.cc:2781] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:37.754767 26891 raft_consensus.cc:482] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:37.754817 26891 raft_consensus.cc:3037] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.755690 26891 raft_consensus.cc:504] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.755945 26891 leader_election.cc:288] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:37.756232 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e148806c0b9143c18ee7938479b481b3" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:37.756345 24897 raft_consensus.cc:3037] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.756522 26666 consensus_queue.cc:237] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:37.757201 24897 raft_consensus.cc:2454] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:37.757227 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e148806c0b9143c18ee7938479b481b3" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:37.757333 24630 raft_consensus.cc:3037] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.758172 24966 leader_election.cc:302] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:37.758267 24630 raft_consensus.cc:2454] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:37.758816 26666 raft_consensus.cc:2781] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:37.758895 26666 raft_consensus.cc:686] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:37.759006 26666 consensus_queue.cc:237] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.759111 24478 catalog_manager.cc:5484] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:37.762375 24478 catalog_manager.cc:5484] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:37.908535 26516 raft_consensus.cc:482] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:37.908681 26516 raft_consensus.cc:504] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.909111 26516 leader_election.cc:288] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:37.909446 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751feb4b2419463c81f3571f6eb0fa42" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:37.909512 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751feb4b2419463c81f3571f6eb0fa42" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:37.909598 24630 raft_consensus.cc:2454] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:37.909621 25034 raft_consensus.cc:2454] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:37.909914 24698 leader_election.cc:302] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:37.910125 26516 raft_consensus.cc:2781] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:37.910194 26516 raft_consensus.cc:482] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:37.910248 26516 raft_consensus.cc:3037] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.911119 26516 raft_consensus.cc:504] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.911463 26516 leader_election.cc:288] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:37.911744 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751feb4b2419463c81f3571f6eb0fa42" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:37.911772 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751feb4b2419463c81f3571f6eb0fa42" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:37.911834 25034 raft_consensus.cc:3037] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.911892 24630 raft_consensus.cc:3037] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:37.912580 25034 raft_consensus.cc:2454] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:37.912837 24630 raft_consensus.cc:2454] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:37.912835 24698 leader_election.cc:302] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:37.913002 26516 raft_consensus.cc:2781] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:37.913125 26516 raft_consensus.cc:686] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:37.913296 26516 consensus_queue.cc:237] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:37.915768 24478 catalog_manager.cc:5484] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:38.078766 26980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:38.078963 26980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:38.081907 26980 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240502 01:58:38.152417 26666 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.154770 25034 raft_consensus.cc:1261] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.154855 24632 raft_consensus.cc:1261] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.155251 26516 consensus_queue.cc:1035] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.156093 26516 consensus_queue.cc:1035] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.157718 26891 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.160666 24631 raft_consensus.cc:1261] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.161088 26891 consensus_queue.cc:1035] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.161317 24897 raft_consensus.cc:1261] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.161798 26891 consensus_queue.cc:1035] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.318 ms
time per row: 0.0516592 ms
I20240502 01:58:38.229247 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:38.234267 24612 tablet_service.cc:1467] Processing CreateTablet for tablet 16ae91433f10465c922183298a387caa (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.234568 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ae91433f10465c922183298a387caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.235133 24876 tablet_service.cc:1467] Processing CreateTablet for tablet d5ec1158a7414039b6f4bc855207c938 (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.235401 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ec1158a7414039b6f4bc855207c938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.235518 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 16ae91433f10465c922183298a387caa (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.235751 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ae91433f10465c922183298a387caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.235782 24744 tablet_service.cc:1467] Processing CreateTablet for tablet d5ec1158a7414039b6f4bc855207c938 (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.236013 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ec1158a7414039b6f4bc855207c938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.237018 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 61ad1a07bc7a4fbcb901792d138e2843 (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.237260 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ad1a07bc7a4fbcb901792d138e2843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.237886 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 61ad1a07bc7a4fbcb901792d138e2843 (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.238183 25012 tablet_service.cc:1467] Processing CreateTablet for tablet d5ec1158a7414039b6f4bc855207c938 (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.235133 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 16ae91433f10465c922183298a387caa (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.238727 27042 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:38.238760 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ae91433f10465c922183298a387caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.239359 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 61ad1a07bc7a4fbcb901792d138e2843 (DEFAULT_TABLE table=rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.239593 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ad1a07bc7a4fbcb901792d138e2843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.239842 27042 tablet_bootstrap.cc:654] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.238180 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ad1a07bc7a4fbcb901792d138e2843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.241083 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ec1158a7414039b6f4bc855207c938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.241293 27042 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:38.241370 27042 ts_tablet_manager.cc:1367] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:38.241947 27041 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:38.241758 27042 raft_consensus.cc:348] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.242048 27042 raft_consensus.cc:374] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.242084 27042 raft_consensus.cc:729] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.242230 27042 consensus_queue.cc:260] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.242738 27041 tablet_bootstrap.cc:654] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.243288 27039 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:38.243427 27042 ts_tablet_manager.cc:1398] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:38.243923 27042 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:38.244024 27039 tablet_bootstrap.cc:654] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.244540 27044 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:38.244810 27042 tablet_bootstrap.cc:654] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.244935 27039 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:38.245023 27039 ts_tablet_manager.cc:1367] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:38.245389 27039 raft_consensus.cc:348] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.245488 27039 raft_consensus.cc:374] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.245519 27039 raft_consensus.cc:729] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.245715 27039 consensus_queue.cc:260] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.246114 27039 ts_tablet_manager.cc:1398] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:38.246263 27039 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:38.246721 27041 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:38.246793 27041 ts_tablet_manager.cc:1367] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:38.246803 27042 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:38.246889 27042 ts_tablet_manager.cc:1367] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:38.247066 27039 tablet_bootstrap.cc:654] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.247167 27041 raft_consensus.cc:348] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.247268 27041 raft_consensus.cc:374] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.247295 27041 raft_consensus.cc:729] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.247293 27042 raft_consensus.cc:348] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.247421 27042 raft_consensus.cc:374] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.247460 27042 raft_consensus.cc:729] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.247421 27041 consensus_queue.cc:260] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.247596 27042 consensus_queue.cc:260] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.247694 27041 ts_tablet_manager.cc:1398] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:38.247844 27041 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:38.247890 27042 ts_tablet_manager.cc:1398] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:38.248374 27044 tablet_bootstrap.cc:654] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.248585 27041 tablet_bootstrap.cc:654] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.249620 27044 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:38.249706 27044 ts_tablet_manager.cc:1367] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:38.250149 27044 raft_consensus.cc:348] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.250241 27044 raft_consensus.cc:374] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.250241 27041 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:38.250274 27044 raft_consensus.cc:729] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.250313 27041 ts_tablet_manager.cc:1367] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:38.250376 27044 consensus_queue.cc:260] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.250581 27044 ts_tablet_manager.cc:1398] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:38.250717 27044 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:38.250676 27041 raft_consensus.cc:348] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.250761 27041 raft_consensus.cc:374] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.250792 27041 raft_consensus.cc:729] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.250895 27041 consensus_queue.cc:260] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.251155 27041 ts_tablet_manager.cc:1398] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:38.251282 27041 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:38.251478 27044 tablet_bootstrap.cc:654] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.252032 27041 tablet_bootstrap.cc:654] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.252851 27044 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:38.252938 27044 ts_tablet_manager.cc:1367] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:38.253319 27044 raft_consensus.cc:348] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.253415 27044 raft_consensus.cc:374] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.253454 27044 raft_consensus.cc:729] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.253559 27044 consensus_queue.cc:260] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.253823 27044 ts_tablet_manager.cc:1398] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:38.253897 27041 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:38.254004 27041 ts_tablet_manager.cc:1367] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:38.254249 27039 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:38.254319 27039 ts_tablet_manager.cc:1367] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:58:38.254396 27041 raft_consensus.cc:348] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.254463 27041 raft_consensus.cc:374] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.254482 27041 raft_consensus.cc:729] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.254573 27041 consensus_queue.cc:260] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.254796 27041 ts_tablet_manager.cc:1398] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:38.254789 27039 raft_consensus.cc:348] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.254878 27039 raft_consensus.cc:374] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.254911 27039 raft_consensus.cc:729] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.255013 27039 consensus_queue.cc:260] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.255298 27039 ts_tablet_manager.cc:1398] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:38.274847 27017 raft_consensus.cc:482] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:38.274932 27017 raft_consensus.cc:504] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.275269 27017 leader_election.cc:288] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:38.275571 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ad1a07bc7a4fbcb901792d138e2843" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:38.275597 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ad1a07bc7a4fbcb901792d138e2843" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:38.275671 24897 raft_consensus.cc:2454] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:38.275709 24630 raft_consensus.cc:2454] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:38.275976 24966 leader_election.cc:302] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:38.276152 27017 raft_consensus.cc:2781] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:38.276207 27017 raft_consensus.cc:482] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:38.276243 27017 raft_consensus.cc:3037] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.277062 27017 raft_consensus.cc:504] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.277340 27017 leader_election.cc:288] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:38.277640 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ad1a07bc7a4fbcb901792d138e2843" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:38.277740 24630 raft_consensus.cc:3037] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.277729 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ad1a07bc7a4fbcb901792d138e2843" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:38.277830 24897 raft_consensus.cc:3037] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.278681 24897 raft_consensus.cc:2454] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:38.278697 24630 raft_consensus.cc:2454] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:38.278983 24965 leader_election.cc:302] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:38.279244 26891 raft_consensus.cc:2781] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:38.279349 26891 raft_consensus.cc:686] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:38.279501 26891 consensus_queue.cc:237] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:38.281376 24478 catalog_manager.cc:5484] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:38.363411 26942 raft_consensus.cc:482] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:38.363528 26942 raft_consensus.cc:504] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.364051 26942 leader_election.cc:288] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:38.364379 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ae91433f10465c922183298a387caa" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:38.364404 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ae91433f10465c922183298a387caa" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:38.364526 24765 raft_consensus.cc:2454] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:38.364535 24630 raft_consensus.cc:2454] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:38.364840 24832 leader_election.cc:302] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:38.365077 26942 raft_consensus.cc:2781] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:38.365142 26942 raft_consensus.cc:482] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:38.365181 26942 raft_consensus.cc:3037] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.366207 26942 raft_consensus.cc:504] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.366547 26942 leader_election.cc:288] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:38.366788 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ae91433f10465c922183298a387caa" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:38.366883 24630 raft_consensus.cc:3037] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.366925 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ae91433f10465c922183298a387caa" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:38.367049 24765 raft_consensus.cc:3037] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.367622 24630 raft_consensus.cc:2454] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:38.367831 24765 raft_consensus.cc:2454] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:38.367862 24832 leader_election.cc:302] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:38.368103 26942 raft_consensus.cc:2781] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:38.368223 26942 raft_consensus.cc:686] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:38.368371 26942 consensus_queue.cc:237] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.370407 24478 catalog_manager.cc:5484] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:38.399772 26942 raft_consensus.cc:482] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:38.399895 26942 raft_consensus.cc:504] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.400243 26942 leader_election.cc:288] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:38.400592 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:38.400734 25032 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:38.400681 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:38.400839 24765 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:38.401067 24830 leader_election.cc:302] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:38.401259 26942 raft_consensus.cc:2781] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:38.401310 26942 raft_consensus.cc:482] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:38.401345 26942 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.402175 26942 raft_consensus.cc:504] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.402491 26942 leader_election.cc:288] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:38.402797 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:38.402904 25032 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.402856 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:38.403014 24765 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.403653 25032 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:38.403935 24765 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:38.403946 24830 leader_election.cc:302] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:38.404116 26942 raft_consensus.cc:2781] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:38.404209 26942 raft_consensus.cc:686] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:38.404493 26942 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:38.406440 24478 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:38.536975 27056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:38.537214 27056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:38.540148 27056 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240502 01:58:38.618108 25032 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.618566 26942 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.618892 24630 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.619553 24764 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.623423 24897 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.624343 26666 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.626562 26942 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.627146 24764 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.627194 24630 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:38.628087 26891 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.633008 26950 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:38.635511 26950 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.231 ms
time per row: 0.0556157 ms
I20240502 01:58:38.690640 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.5:40239
--local_ip_for_outbound_sockets=127.17.243.5
--tserver_master_addrs=127.17.243.62:33801
--webserver_port=42201
--webserver_interface=127.17.243.5
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:58:38.800491 27095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:38.800721 27095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:38.800791 27095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:38.803716 27095 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:38.803776 27095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:38.803853 27095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.5
I20240502 01:58:38.807585 27095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.5:40239
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.243.5
--webserver_port=42201
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.27095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.5
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:38.808401 27095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:38.809437 27095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:38.818207 27103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:38.818564 27100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:38.819684 27095 server_base.cc:1034] running on GCE node
W20240502 01:58:38.820197 27101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:38.820672 27095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:38.821614 27095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:38.822839 27095 hybrid_clock.cc:648] HybridClock initialized: now 1714615118822792 us; error 62 us; skew 500 ppm
I20240502 01:58:38.824579 27095 webserver.cc:460] Webserver started at http://127.17.243.5:42201/ using document root <none> and password file <none>
I20240502 01:58:38.825078 27095 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:38.825150 27095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:38.828267 27095 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:38.830109 27109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:38.831102 27095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:38.831208 27095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:38.831555 27095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:38.864228 27095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:38.864874 27095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:38.865028 27095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:38.865594 27095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:38.866717 27095 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:38.866781 27095 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:38.866840 27095 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:38.866871 27095 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:38.877126 27095 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.5:40239
I20240502 01:58:38.877346 27224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.5:40239 every 8 connection(s)
I20240502 01:58:38.878466 27095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:38.886248 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 27095
I20240502 01:58:38.886744 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.6:41707
--local_ip_for_outbound_sockets=127.17.243.6
--tserver_master_addrs=127.17.243.62:33801
--webserver_port=40299
--webserver_interface=127.17.243.6
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:38.889690 27225 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:38.889976 27225 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:38.890657 27225 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:38.891824 24478 ts_manager.cc:194] Re-registered known tserver with Master: 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:38.892391 24478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.5:35247
W20240502 01:58:39.043877 27228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:39.044095 27228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:39.044139 27228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:39.047247 27228 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:39.047307 27228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:39.047372 27228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.6
I20240502 01:58:39.050977 27228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.6:41707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.17.243.6
--webserver_port=40299
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.27228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.6
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:39.051831 27228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:39.052913 27228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:39.061314 27234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:39.061589 27233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:39.061942 27236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:39.062547 27228 server_base.cc:1034] running on GCE node
I20240502 01:58:39.062871 27228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:39.063344 27228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:39.064478 27228 hybrid_clock.cc:648] HybridClock initialized: now 1714615119064447 us; error 44 us; skew 500 ppm
I20240502 01:58:39.066300 27228 webserver.cc:460] Webserver started at http://127.17.243.6:40299/ using document root <none> and password file <none>
I20240502 01:58:39.066800 27228 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:39.066865 27228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:39.070124 27228 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:39.071942 27242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.072870 27228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:39.072959 27228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1049b4c8783b48f39be22bd113ad349f"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:39.073329 27228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:39.085476 27228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:39.086072 27228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:39.086227 27228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:39.086704 27228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:39.087489 27228 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:39.087545 27228 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.087603 27228 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:39.087626 27228 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.095651 27228 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.6:41707
I20240502 01:58:39.095707 27357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.6:41707 every 8 connection(s)
I20240502 01:58:39.096585 27228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:39.105116 27358 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:39.105306 27358 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:39.105680 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 27228
I20240502 01:58:39.105783 27358 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:39.106125 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.7:42021
--local_ip_for_outbound_sockets=127.17.243.7
--tserver_master_addrs=127.17.243.62:33801
--webserver_port=39385
--webserver_interface=127.17.243.7
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:39.106812 24478 ts_manager.cc:194] Re-registered known tserver with Master: 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:39.107575 24478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.6:42365
W20240502 01:58:39.217170 27361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:39.217353 27361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:39.217407 27361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:39.220311 27361 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:39.220372 27361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:39.220451 27361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.7
I20240502 01:58:39.223976 27361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.7:42021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.17.243.7
--webserver_port=39385
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.27361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.7
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:39.224731 27361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:39.225732 27361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:39.233762 27368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:39.234043 27367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:39.234596 27370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:39.235095 27361 server_base.cc:1034] running on GCE node
I20240502 01:58:39.235483 27361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:39.235970 27361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:39.237110 27361 hybrid_clock.cc:648] HybridClock initialized: now 1714615119237099 us; error 23 us; skew 500 ppm
I20240502 01:58:39.239054 27361 webserver.cc:460] Webserver started at http://127.17.243.7:39385/ using document root <none> and password file <none>
I20240502 01:58:39.239521 27361 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:39.239580 27361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:39.242619 27361 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:39.244342 27376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.245242 27361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:39.245330 27361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:39.245682 27361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:39.265166 27361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:39.265643 27361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:39.265761 27361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:39.266222 27361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:39.267016 27361 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:39.267060 27361 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.267095 27361 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:39.267115 27361 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.276703 27361 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.7:42021
I20240502 01:58:39.276731 27491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.7:42021 every 8 connection(s)
I20240502 01:58:39.277704 27361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:39.282850 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 27361
I20240502 01:58:39.283205 18380 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5aC8CU/build/debug/bin/kudu
/tmp/dist-test-task5aC8CU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.243.8:44673
--local_ip_for_outbound_sockets=127.17.243.8
--tserver_master_addrs=127.17.243.62:33801
--webserver_port=45521
--webserver_interface=127.17.243.8
--builtin_ntp_servers=127.17.243.20:46399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:58:39.286674 27492 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:39.286887 27492 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:39.287400 27492 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:39.288472 24478 ts_manager.cc:194] Re-registered known tserver with Master: 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:39.289197 24478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.7:52311
W20240502 01:58:39.391359 27495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:39.391557 27495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:39.391599 27495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:39.394513 27495 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:39.394572 27495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:39.394636 27495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.243.8
I20240502 01:58:39.398270 27495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.243.20:46399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.243.8:44673
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.17.243.8
--webserver_port=45521
--tserver_master_addrs=127.17.243.62:33801
--never_fsync=true
--heap_profile_path=/tmp/kudu.27495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.243.8
--log_dir=/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:39.399082 27495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:39.400125 27495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:39.407979 27503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:39.408285 27500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:39.408854 27501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:39.409356 27495 server_base.cc:1034] running on GCE node
I20240502 01:58:39.409698 27495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:39.410207 27495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:39.411343 27495 hybrid_clock.cc:648] HybridClock initialized: now 1714615119411329 us; error 27 us; skew 500 ppm
I20240502 01:58:39.413276 27495 webserver.cc:460] Webserver started at http://127.17.243.8:45521/ using document root <none> and password file <none>
I20240502 01:58:39.413893 27495 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:39.414012 27495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:39.417330 27495 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:39.419785 27509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.420732 27495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:39.420831 27495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bb653087e8554c7591cf547dd969f3be"
format_stamp: "Formatted at 2024-05-02 01:58:23 on dist-test-slave-j3s5"
I20240502 01:58:39.421190 27495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:39.446043 27495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:39.446722 27495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:39.446885 27495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:39.447499 27495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:39.448534 27495 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:39.448614 27495 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.448676 27495 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:39.448704 27495 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:39.458289 27495 rpc_server.cc:307] RPC server started. Bound to: 127.17.243.8:44673
I20240502 01:58:39.458336 27624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.243.8:44673 every 8 connection(s)
I20240502 01:58:39.459611 27495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:58:39.468609 27625 heartbeater.cc:347] Connected to a master server at 127.17.243.62:33801
I20240502 01:58:39.468814 27625 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:39.469343 27625 heartbeater.cc:511] Master 127.17.243.62:33801 requested a full tablet report, sending...
I20240502 01:58:39.469914 18380 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5aC8CU/build/debug/bin/kudu as pid 27495
I20240502 01:58:39.470399 24478 ts_manager.cc:194] Re-registered known tserver with Master: bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:39.471141 24478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.243.8:49355
I20240502 01:58:39.473404 24478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:39.480355 24877 tablet_service.cc:1467] Processing CreateTablet for tablet c477d70d401942c8a0f3972c313f81ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]), partition=
I20240502 01:58:39.480643 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477d70d401942c8a0f3972c313f81ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.483129 27641 tablet_bootstrap.cc:492] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:39.484167 27641 tablet_bootstrap.cc:654] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.487608 27641 tablet_bootstrap.cc:492] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:39.487691 27641 ts_tablet_manager.cc:1367] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:39.488076 27641 raft_consensus.cc:348] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.488176 27641 raft_consensus.cc:374] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.488201 27641 raft_consensus.cc:729] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.488318 27641 consensus_queue.cc:260] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.488345 27290 tablet_service.cc:1467] Processing CreateTablet for tablet c477d70d401942c8a0f3972c313f81ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]), partition=
I20240502 01:58:39.488610 27641 ts_tablet_manager.cc:1398] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.489289 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477d70d401942c8a0f3972c313f81ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.491747 27157 tablet_service.cc:1467] Processing CreateTablet for tablet c477d70d401942c8a0f3972c313f81ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]), partition=
I20240502 01:58:39.492446 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477d70d401942c8a0f3972c313f81ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.500089 27643 tablet_bootstrap.cc:492] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:39.502180 27644 tablet_bootstrap.cc:492] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:39.502611 27643 tablet_bootstrap.cc:654] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.503648 27643 log.cc:824] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:39.504482 27644 tablet_bootstrap.cc:654] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.505295 27644 log.cc:824] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:39.507103 27643 tablet_bootstrap.cc:492] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:39.507138 27644 tablet_bootstrap.cc:492] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:39.507280 27644 ts_tablet_manager.cc:1367] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:58:39.507280 27643 ts_tablet_manager.cc:1367] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:58:39.511001 27643 raft_consensus.cc:348] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.511241 27643 raft_consensus.cc:374] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.511304 27643 raft_consensus.cc:729] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.511605 27644 raft_consensus.cc:348] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.511782 27644 raft_consensus.cc:374] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.511729 27643 consensus_queue.cc:260] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.511832 27644 raft_consensus.cc:729] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.512264 27644 consensus_queue.cc:260] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.512401 27225 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
I20240502 01:58:39.512549 27643 ts_tablet_manager.cc:1398] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 01:58:39.513064 27644 ts_tablet_manager.cc:1398] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240502 01:58:39.513353 27358 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
W20240502 01:58:39.597972 27359 tablet.cc:2369] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:39.631096 27226 tablet.cc:2369] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:39.818114 27649 raft_consensus.cc:482] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:39.818301 27649 raft_consensus.cc:504] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.819408 27649 leader_election.cc:288] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:39.822715 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d70d401942c8a0f3972c313f81ca" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:39.822844 24896 raft_consensus.cc:2454] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:39.822913 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d70d401942c8a0f3972c313f81ca" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:39.823206 27177 raft_consensus.cc:2454] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:39.823197 27246 leader_election.cc:302] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:39.823431 27649 raft_consensus.cc:2781] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:39.823511 27649 raft_consensus.cc:482] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:39.823567 27649 raft_consensus.cc:3037] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.824491 27649 raft_consensus.cc:504] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.824815 27649 leader_election.cc:288] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:39.825152 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d70d401942c8a0f3972c313f81ca" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:39.825245 24896 raft_consensus.cc:3037] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.825229 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d70d401942c8a0f3972c313f81ca" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:39.825350 27177 raft_consensus.cc:3037] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.826098 24896 raft_consensus.cc:2454] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:39.826524 27177 raft_consensus.cc:2454] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:39.826835 27245 leader_election.cc:302] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:39.827204 27649 raft_consensus.cc:2781] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:39.827411 27649 raft_consensus.cc:686] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:39.827709 27649 consensus_queue.cc:237] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:39.830273 24482 catalog_manager.cc:5484] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:39.861277 27290 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:39.862038 27424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:39.866300 24610 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:58:39.866575 27157 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:39.866858 24743 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:58:39.867398 27555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:39.888571 24877 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:58:39.890014 25011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:58:39.947096 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:39.953338 24897 raft_consensus.cc:1261] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:39.954180 27694 consensus_queue.cc:1035] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:39.954768 27177 raft_consensus.cc:1261] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:39.955293 27694 consensus_queue.cc:1035] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:39.959496 24482 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]). Expected version 1 got 0
I20240502 01:58:39.960312 27731 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:39.961163 27731 tablet.cc:1620] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:39.962287 24482 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]). Expected version 1 got 0
I20240502 01:58:39.965139 24484 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]). Expected version 1 got 0
I20240502 01:58:39.965797 27734 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:39.966532 27734 tablet.cc:1620] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:39.966899 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]). Expected version 1 got 0
I20240502 01:58:39.967344 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091] alter complete (version 1)
I20240502 01:58:39.971632 24484 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:39.972034 24479 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091]). Expected version 1 got 0
I20240502 01:58:39.973618 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 01:58:39.973728 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 01:58:39.976632 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet e96b29005a51413082c19a8c79698334
I20240502 01:58:39.977239 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet e96b29005a51413082c19a8c79698334 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:39.977387 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 1a3c0e77bfb04e71a5dd553245353cfa
I20240502 01:58:39.977638 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet e96b29005a51413082c19a8c79698334 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:39.977807 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 65f16854b1544fb7928482ce0a2f9921
I20240502 01:58:39.977977 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet e96b29005a51413082c19a8c79698334 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:39.978022 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3c0e77bfb04e71a5dd553245353cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:39.978394 27741 tablet_replica.cc:330] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:39.978694 27741 raft_consensus.cc:2229] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:39.978850 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f16854b1544fb7928482ce0a2f9921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:39.979054 27741 raft_consensus.cc:2258] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:39.979254 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f16854b1544fb7928482ce0a2f9921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:39.979307 27737 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:39.979959 27737 tablet.cc:1620] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:39.980034 27741 ts_tablet_manager.cc:1875] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:39.980285 27739 tablet_replica.cc:330] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:39.980569 27739 raft_consensus.cc:2229] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:39.981092 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2436f6b4344a2294be04f84f52f5ff (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.981390 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 2078a814ae1648e2bde5e521ef45af6d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.982362 27742 tablet_replica.cc:330] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:39.982630 27742 raft_consensus.cc:2229] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:39.982908 27742 raft_consensus.cc:2258] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:39.983384 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3c0e77bfb04e71a5dd553245353cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:39.983793 27742 ts_tablet_manager.cc:1875] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:39.984134 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2436f6b4344a2294be04f84f52f5ff (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.984421 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2436f6b4344a2294be04f84f52f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.985903 27742 ts_tablet_manager.cc:1888] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:39.985998 27742 log.cc:1196] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a3c0e77bfb04e71a5dd553245353cfa
I20240502 01:58:39.986050 27741 ts_tablet_manager.cc:1888] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:39.986132 27741 log.cc:1196] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/e96b29005a51413082c19a8c79698334
I20240502 01:58:39.986266 27742 ts_tablet_manager.cc:1909] T 1a3c0e77bfb04e71a5dd553245353cfa P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:39.987166 27742 tablet_replica.cc:330] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:39.987286 27742 raft_consensus.cc:2229] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:39.987454 27742 raft_consensus.cc:2258] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:39.987910 27742 ts_tablet_manager.cc:1875] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:39.988078 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 823fe241dc1b40cca121a22d669e4226 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.988301 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823fe241dc1b40cca121a22d669e4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.988399 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 2078a814ae1648e2bde5e521ef45af6d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.989382 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2078a814ae1648e2bde5e521ef45af6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.989553 27742 ts_tablet_manager.cc:1888] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:39.989609 27742 log.cc:1196] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/65f16854b1544fb7928482ce0a2f9921
I20240502 01:58:39.989832 27742 ts_tablet_manager.cc:1909] T 65f16854b1544fb7928482ce0a2f9921 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:39.987684 24612 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f16854b1544fb7928482ce0a2f9921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:39.990087 27643 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:39.988615 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 823fe241dc1b40cca121a22d669e4226 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.990247 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 1a3c0e77bfb04e71a5dd553245353cfa (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:39.990319 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823fe241dc1b40cca121a22d669e4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.990523 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2436f6b4344a2294be04f84f52f5ff (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.990715 27739 raft_consensus.cc:2258] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:39.990794 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2436f6b4344a2294be04f84f52f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.990885 27643 tablet_bootstrap.cc:654] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.992255 27643 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:39.992323 27643 ts_tablet_manager.cc:1367] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:39.992496 27739 ts_tablet_manager.cc:1875] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:39.988760 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3c0e77bfb04e71a5dd553245353cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:39 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:39.992862 27643 raft_consensus.cc:348] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:39.992964 27643 raft_consensus.cc:374] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.992995 27643 raft_consensus.cc:729] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.993083 27643 consensus_queue.cc:260] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:39.993305 27643 ts_tablet_manager.cc:1398] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.994187 27740 tablet_replica.cc:330] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:39.994227 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 65f16854b1544fb7928482ce0a2f9921 (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:39.981447 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2436f6b4344a2294be04f84f52f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.995435 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2078a814ae1648e2bde5e521ef45af6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.995463 27739 ts_tablet_manager.cc:1888] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:39.995549 27739 log.cc:1196] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/e96b29005a51413082c19a8c79698334
I20240502 01:58:39.995904 27739 ts_tablet_manager.cc:1909] T e96b29005a51413082c19a8c79698334 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:39.997301 27739 tablet_replica.cc:330] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:39.997509 27739 raft_consensus.cc:2229] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:39.995322 27740 raft_consensus.cc:2229] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:39.997596 27644 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:39.997705 27739 raft_consensus.cc:2258] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:39.997848 27740 raft_consensus.cc:2258] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:39.998363 27739 ts_tablet_manager.cc:1875] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:39.998531 27644 tablet_bootstrap.cc:654] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.998950 27740 ts_tablet_manager.cc:1875] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.000138 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet e96b29005a51413082c19a8c79698334 (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.000548 27739 ts_tablet_manager.cc:1888] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.000900 27739 log.cc:1196] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/65f16854b1544fb7928482ce0a2f9921
I20240502 01:58:40.001323 27741 ts_tablet_manager.cc:1909] T e96b29005a51413082c19a8c79698334 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:40.001542 27739 ts_tablet_manager.cc:1909] T 65f16854b1544fb7928482ce0a2f9921 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:40.002141 27739 tablet_replica.cc:330] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:40.002297 27740 ts_tablet_manager.cc:1888] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.002323 27739 raft_consensus.cc:2229] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.002446 27739 raft_consensus.cc:2258] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.002460 27740 log.cc:1196] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/e96b29005a51413082c19a8c79698334
I20240502 01:58:40.002751 27740 ts_tablet_manager.cc:1909] T e96b29005a51413082c19a8c79698334 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:40.002723 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet e96b29005a51413082c19a8c79698334 (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.002883 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 65f16854b1544fb7928482ce0a2f9921 (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.002959 27741 tablet_replica.cc:330] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:40.003051 27739 ts_tablet_manager.cc:1875] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.004029 27740 tablet_replica.cc:330] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:40.004213 27740 raft_consensus.cc:2229] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.004374 27740 raft_consensus.cc:2258] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.003175 27741 raft_consensus.cc:2229] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.004657 27741 raft_consensus.cc:2258] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.005050 27739 ts_tablet_manager.cc:1888] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.005177 27739 log.cc:1196] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a3c0e77bfb04e71a5dd553245353cfa
I20240502 01:58:40.005263 27741 ts_tablet_manager.cc:1875] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.005538 27739 ts_tablet_manager.cc:1909] T 1a3c0e77bfb04e71a5dd553245353cfa P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:40.005540 27740 ts_tablet_manager.cc:1875] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.006541 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 1a3c0e77bfb04e71a5dd553245353cfa (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.006951 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 2078a814ae1648e2bde5e521ef45af6d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.007864 27741 ts_tablet_manager.cc:1888] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.007953 27741 log.cc:1196] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/1a3c0e77bfb04e71a5dd553245353cfa
I20240502 01:58:40.008008 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2078a814ae1648e2bde5e521ef45af6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.008211 27741 ts_tablet_manager.cc:1909] T 1a3c0e77bfb04e71a5dd553245353cfa P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:40.008430 27740 ts_tablet_manager.cc:1888] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.008500 27740 log.cc:1196] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/65f16854b1544fb7928482ce0a2f9921
I20240502 01:58:40.008711 27740 ts_tablet_manager.cc:1909] T 65f16854b1544fb7928482ce0a2f9921 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:40.008942 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 1a3c0e77bfb04e71a5dd553245353cfa (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.009137 27746 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:40.010306 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet e96b29005a51413082c19a8c79698334 (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.010464 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 65f16854b1544fb7928482ce0a2f9921 (table rebalance_test_table_0 [id=957e8890b65645d081c26680147f5f14]) successfully deleted
I20240502 01:58:40.006958 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 823fe241dc1b40cca121a22d669e4226 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.010761 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823fe241dc1b40cca121a22d669e4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.011307 27746 tablet_bootstrap.cc:654] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.011997 27746 log.cc:824] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:40.013610 27746 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:40.013849 27746 ts_tablet_manager.cc:1367] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:40.014899 27643 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:40.014945 27749 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:40.015858 27643 tablet_bootstrap.cc:654] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.016777 27749 tablet_bootstrap.cc:654] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.016767 27746 raft_consensus.cc:348] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.016978 27746 raft_consensus.cc:374] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.017004 27643 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:40.017040 27746 raft_consensus.cc:729] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.017107 27643 ts_tablet_manager.cc:1367] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:40.017441 27749 log.cc:824] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:40.017539 27746 consensus_queue.cc:260] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.017621 27643 raft_consensus.cc:348] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.017730 27643 raft_consensus.cc:374] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.017764 27643 raft_consensus.cc:729] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.017901 27643 consensus_queue.cc:260] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.018200 27643 ts_tablet_manager.cc:1398] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.018232 27492 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
I20240502 01:58:40.018426 27746 ts_tablet_manager.cc:1398] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:40.018440 27744 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:40.019305 27746 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:40.020268 27644 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:40.020363 27644 ts_tablet_manager.cc:1367] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 01:58:40.020574 27744 tablet_bootstrap.cc:654] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.020385 27746 tablet_bootstrap.cc:654] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.020922 27644 raft_consensus.cc:348] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.021023 27644 raft_consensus.cc:374] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.021054 27644 raft_consensus.cc:729] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.021179 27644 consensus_queue.cc:260] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.021430 27644 ts_tablet_manager.cc:1398] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.021587 27644 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:40.022442 27644 tablet_bootstrap.cc:654] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.023173 27746 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:40.023262 27746 ts_tablet_manager.cc:1367] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:40.023799 27746 raft_consensus.cc:348] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.023900 27746 raft_consensus.cc:374] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.023932 27746 raft_consensus.cc:729] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.024114 27746 consensus_queue.cc:260] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.024396 27746 ts_tablet_manager.cc:1398] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.024504 27644 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:40.024585 27644 ts_tablet_manager.cc:1367] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:40.024967 27644 raft_consensus.cc:348] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.025089 27644 raft_consensus.cc:374] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.025130 27644 raft_consensus.cc:729] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.025244 27644 consensus_queue.cc:260] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.025480 27644 ts_tablet_manager.cc:1398] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.026034 27749 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:40.026217 27749 ts_tablet_manager.cc:1367] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:58:40.028563 27744 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:40.028697 27744 ts_tablet_manager.cc:1367] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
W20240502 01:58:40.029021 27493 tablet.cc:2369] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:40.029114 27744 raft_consensus.cc:348] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.029206 27744 raft_consensus.cc:374] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.029234 27744 raft_consensus.cc:729] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.029346 27744 consensus_queue.cc:260] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.029695 27744 ts_tablet_manager.cc:1398] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.029662 27749 raft_consensus.cc:348] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.029856 27749 raft_consensus.cc:374] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.029944 27749 raft_consensus.cc:729] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.031010 27749 consensus_queue.cc:260] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.031829 27749 ts_tablet_manager.cc:1398] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:58:40.032156 27625 heartbeater.cc:503] Master 127.17.243.62:33801 was elected leader, sending a full tablet report...
I20240502 01:58:40.032152 27749 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:40.033396 27749 tablet_bootstrap.cc:654] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.035432 27749 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:40.035521 27749 ts_tablet_manager.cc:1367] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:40.035962 27749 raft_consensus.cc:348] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.036068 27749 raft_consensus.cc:374] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.036094 27749 raft_consensus.cc:729] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.036206 27749 consensus_queue.cc:260] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.036434 27749 ts_tablet_manager.cc:1398] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.057281 27648 raft_consensus.cc:482] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.057389 27648 raft_consensus.cc:504] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.058848 27648 leader_election.cc:288] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:40.062070 27766 raft_consensus.cc:482] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.062222 27766 raft_consensus.cc:504] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.063246 27766 leader_election.cc:288] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:40.068061 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823fe241dc1b40cca121a22d669e4226" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:40.068202 27177 raft_consensus.cc:2454] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:40.069182 27513 leader_election.cc:302] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:40.069592 27766 raft_consensus.cc:2781] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.069664 27766 raft_consensus.cc:482] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.069705 27766 raft_consensus.cc:3037] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.070837 27766 raft_consensus.cc:504] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.071173 27766 leader_election.cc:288] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:40.072651 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823fe241dc1b40cca121a22d669e4226" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:40.072687 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823fe241dc1b40cca121a22d669e4226" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:40.072662 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823fe241dc1b40cca121a22d669e4226" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:40.072755 27177 raft_consensus.cc:3037] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.072796 24632 raft_consensus.cc:3037] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.073571 24632 raft_consensus.cc:2454] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:40.073699 27177 raft_consensus.cc:2454] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:40.074086 27513 leader_election.cc:302] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:40.074628 27766 raft_consensus.cc:2781] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.074776 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:40.074867 27766 raft_consensus.cc:686] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:40.075610 27766 consensus_queue.cc:237] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.077687 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823fe241dc1b40cca121a22d669e4226" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:40.080403 24478 catalog_manager.cc:5484] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:40.081676 27731 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:40.082005 27731 tablet.cc:1620] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:40.082639 27734 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:40.082901 27734 tablet.cc:1620] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:40.084241 27737 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:40.084527 27737 tablet.cc:1620] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:40.087116 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823fe241dc1b40cca121a22d669e4226" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:40.087278 24632 raft_consensus.cc:2379] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dd619a595e94004ac0f7f5a9054e7cc in current term 1: Already voted for candidate bb653087e8554c7591cf547dd969f3be in this term.
I20240502 01:58:40.087610 27112 leader_election.cc:302] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc; no voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be
I20240502 01:58:40.088106 27648 raft_consensus.cc:2726] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:58:40.087894 24478 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5c2ee56efa6740ebba5ca767f1186091] alter complete (version 2)
I20240502 01:58:40.193578 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:40.199505 27731 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:40.200196 27737 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:40.200201 27734 tablet.cc:1722] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:40.204690 24478 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=5c2ee56efa6740ebba5ca767f1186091] alter complete (version 3)
W20240502 01:58:40.210925 27626 tablet.cc:2369] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:40.274780 27736 raft_consensus.cc:482] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.274899 27736 raft_consensus.cc:504] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.275349 27736 leader_election.cc:288] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:40.279268 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:40.279549 27446 raft_consensus.cc:2454] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:40.280020 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:40.280045 27246 leader_election.cc:302] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:40.280194 27578 raft_consensus.cc:2454] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:40.280220 27736 raft_consensus.cc:2781] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.280267 27736 raft_consensus.cc:482] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.280300 27736 raft_consensus.cc:3037] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.281257 27736 raft_consensus.cc:504] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.281586 27736 leader_election.cc:288] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:40.281924 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:40.282039 27578 raft_consensus.cc:3037] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.282967 27578 raft_consensus.cc:2454] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:40.283224 27246 leader_election.cc:302] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:40.283589 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:40.283658 27736 raft_consensus.cc:2781] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.283722 27446 raft_consensus.cc:3037] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.283748 27736 raft_consensus.cc:686] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:40.283874 27736 consensus_queue.cc:237] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:40.284610 27446 raft_consensus.cc:2454] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:40.285501 24482 catalog_manager.cc:5484] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:40.289469 27648 raft_consensus.cc:482] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.289566 27648 raft_consensus.cc:504] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.289975 27648 leader_election.cc:288] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:40.293320 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2436f6b4344a2294be04f84f52f5ff" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:40.293419 27446 raft_consensus.cc:2454] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:40.293504 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2436f6b4344a2294be04f84f52f5ff" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:40.293692 27312 raft_consensus.cc:2454] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:40.293684 27113 leader_election.cc:302] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:40.293841 27648 raft_consensus.cc:2781] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.293900 27648 raft_consensus.cc:482] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.293959 27648 raft_consensus.cc:3037] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.294740 27648 raft_consensus.cc:504] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.295063 27648 leader_election.cc:288] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:40.295346 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2436f6b4344a2294be04f84f52f5ff" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:40.295374 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2436f6b4344a2294be04f84f52f5ff" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:40.295456 27312 raft_consensus.cc:3037] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.295464 27446 raft_consensus.cc:3037] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.296263 27446 raft_consensus.cc:2454] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:40.296289 27312 raft_consensus.cc:2454] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:40.296571 27113 leader_election.cc:302] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:40.296762 27648 raft_consensus.cc:2781] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.296936 27648 raft_consensus.cc:686] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:40.297204 27648 consensus_queue.cc:237] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:40.299479 24482 catalog_manager.cc:5484] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.307497 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:40.307593 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:40.309334 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet c477d70d401942c8a0f3972c313f81ca
I20240502 01:58:40.309859 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet c477d70d401942c8a0f3972c313f81ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:40.309965 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet c477d70d401942c8a0f3972c313f81ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:40.310372 27800 tablet_replica.cc:330] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:40.310417 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet c477d70d401942c8a0f3972c313f81ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:40.310604 27741 tablet_replica.cc:330] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:40.310653 27799 tablet_replica.cc:330] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:40.310736 27741 raft_consensus.cc:2229] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.310842 27741 raft_consensus.cc:2258] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.310995 27799 raft_consensus.cc:2229] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.311465 27741 ts_tablet_manager.cc:1875] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.313549 27741 ts_tablet_manager.cc:1888] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:40.313624 27741 log.cc:1196] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/c477d70d401942c8a0f3972c313f81ca
I20240502 01:58:40.313859 27741 ts_tablet_manager.cc:1909] T c477d70d401942c8a0f3972c313f81ca P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:40.312953 27799 raft_consensus.cc:2258] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.314889 27800 raft_consensus.cc:2229] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.314983 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table_new_name [id=5c2ee56efa6740ebba5ca767f1186091]) successfully deleted
I20240502 01:58:40.315049 27800 raft_consensus.cc:2258] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.315171 27799 ts_tablet_manager.cc:1875] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.315990 27800 ts_tablet_manager.cc:1875] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.317242 27799 ts_tablet_manager.cc:1888] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:40.317327 27799 log.cc:1196] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/c477d70d401942c8a0f3972c313f81ca
I20240502 01:58:40.317595 27799 ts_tablet_manager.cc:1909] T c477d70d401942c8a0f3972c313f81ca P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:40.318156 27800 ts_tablet_manager.cc:1888] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:40.318228 27800 log.cc:1196] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/c477d70d401942c8a0f3972c313f81ca
I20240502 01:58:40.318535 27800 ts_tablet_manager.cc:1909] T c477d70d401942c8a0f3972c313f81ca P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:40.319525 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table_new_name [id=5c2ee56efa6740ebba5ca767f1186091]) successfully deleted
I20240502 01:58:40.319636 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet c477d70d401942c8a0f3972c313f81ca (table rebalancer_dynamic_table_new_name [id=5c2ee56efa6740ebba5ca767f1186091]) successfully deleted
I20240502 01:58:40.323359 24610 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:58:40.326583 27424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:40.329449 27157 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.343031 24743 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:58:40.345239 27290 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.345494 25011 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:58:40.351615 24877 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:58:40.354140 27555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.411620 24478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:40.418004 27424 tablet_service.cc:1467] Processing CreateTablet for tablet f83fb5ee990d493eba6e78eca7e747bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]), partition=
I20240502 01:58:40.418141 27290 tablet_service.cc:1467] Processing CreateTablet for tablet f83fb5ee990d493eba6e78eca7e747bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]), partition=
I20240502 01:58:40.418267 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83fb5ee990d493eba6e78eca7e747bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.418421 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83fb5ee990d493eba6e78eca7e747bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.420012 27746 tablet_bootstrap.cc:492] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:40.420459 27644 tablet_bootstrap.cc:492] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:40.420756 27746 tablet_bootstrap.cc:654] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.421202 27644 tablet_bootstrap.cc:654] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.421438 24610 tablet_service.cc:1467] Processing CreateTablet for tablet f83fb5ee990d493eba6e78eca7e747bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]), partition=
I20240502 01:58:40.421646 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83fb5ee990d493eba6e78eca7e747bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.422189 27644 tablet_bootstrap.cc:492] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:40.422255 27644 ts_tablet_manager.cc:1367] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:40.422642 27746 tablet_bootstrap.cc:492] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:40.422622 27644 raft_consensus.cc:348] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.422713 27644 raft_consensus.cc:374] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.422717 27746 ts_tablet_manager.cc:1367] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:40.422744 27644 raft_consensus.cc:729] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.422856 27644 consensus_queue.cc:260] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.423125 27644 ts_tablet_manager.cc:1398] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.423115 27746 raft_consensus.cc:348] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.423219 27746 raft_consensus.cc:374] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.423251 27746 raft_consensus.cc:729] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.423379 27746 consensus_queue.cc:260] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.423664 27746 ts_tablet_manager.cc:1398] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.424798 27744 tablet_bootstrap.cc:492] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:40.425480 27744 tablet_bootstrap.cc:654] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.427439 27744 tablet_bootstrap.cc:492] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:40.427522 27744 ts_tablet_manager.cc:1367] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:40.427858 27744 raft_consensus.cc:348] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.427945 27744 raft_consensus.cc:374] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.427973 27744 raft_consensus.cc:729] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.428085 27744 consensus_queue.cc:260] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.428351 27744 ts_tablet_manager.cc:1398] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.479509 24485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 01:58:40.479660 24485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 01:58:40.485359 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 00616380fd3941159bfd4c2a6bba5761
I20240502 01:58:40.486050 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 00616380fd3941159bfd4c2a6bba5761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:40.486099 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet fc67bd4c105d49c3afe3940637db5ab9
I20240502 01:58:40.486352 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 00616380fd3941159bfd4c2a6bba5761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:40.486574 27739 tablet_replica.cc:330] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:40.486703 27739 raft_consensus.cc:2229] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.486825 27739 raft_consensus.cc:2258] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.487215 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet fc67bd4c105d49c3afe3940637db5ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:40.487361 27739 ts_tablet_manager.cc:1875] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.487622 27741 tablet_replica.cc:330] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:40.487757 27741 raft_consensus.cc:2229] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.487872 27741 raft_consensus.cc:2258] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.488122 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 7a51df555e1143b8b8db1d70f7407a1f
I20240502 01:58:40.488364 27741 ts_tablet_manager.cc:1875] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.488533 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a51df555e1143b8b8db1d70f7407a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:40.488919 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a51df555e1143b8b8db1d70f7407a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:40.489540 27739 ts_tablet_manager.cc:1888] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.489593 27739 log.cc:1196] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/00616380fd3941159bfd4c2a6bba5761
I20240502 01:58:40.489763 27739 ts_tablet_manager.cc:1909] T 00616380fd3941159bfd4c2a6bba5761 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:40.489913 27740 tablet_replica.cc:330] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:40.490056 27740 raft_consensus.cc:2229] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.490276 27740 raft_consensus.cc:2258] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.490420 27741 ts_tablet_manager.cc:1888] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.490481 27741 log.cc:1196] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/fc67bd4c105d49c3afe3940637db5ab9
I20240502 01:58:40.490689 27740 ts_tablet_manager.cc:1875] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.490708 27741 ts_tablet_manager.cc:1909] T fc67bd4c105d49c3afe3940637db5ab9 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:40.490986 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 00616380fd3941159bfd4c2a6bba5761 (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.491359 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet fc67bd4c105d49c3afe3940637db5ab9 (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.491201 27739 tablet_replica.cc:330] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:40.491603 27739 raft_consensus.cc:2229] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.491845 27739 raft_consensus.cc:2258] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.492377 27739 ts_tablet_manager.cc:1875] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.492587 27740 ts_tablet_manager.cc:1888] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.492652 27740 log.cc:1196] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/00616380fd3941159bfd4c2a6bba5761
I20240502 01:58:40.492906 27740 ts_tablet_manager.cc:1909] T 00616380fd3941159bfd4c2a6bba5761 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:40.493453 27740 tablet_replica.cc:330] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:40.493583 27740 raft_consensus.cc:2229] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.493695 27740 raft_consensus.cc:2258] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.494454 27739 ts_tablet_manager.cc:1888] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.494526 27739 log.cc:1196] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/7a51df555e1143b8b8db1d70f7407a1f
I20240502 01:58:40.494680 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet fc67bd4c105d49c3afe3940637db5ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:40.494755 27739 ts_tablet_manager.cc:1909] T 7a51df555e1143b8b8db1d70f7407a1f P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:40.494843 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet fc67bd4c105d49c3afe3940637db5ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:40.495272 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a51df555e1143b8b8db1d70f7407a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:40.495282 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 00616380fd3941159bfd4c2a6bba5761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:40.496392 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 00616380fd3941159bfd4c2a6bba5761 (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.497319 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 7a51df555e1143b8b8db1d70f7407a1f (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.497542 27740 ts_tablet_manager.cc:1875] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.497625 27739 tablet_replica.cc:330] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:40.497741 27739 raft_consensus.cc:2229] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.497855 27739 raft_consensus.cc:2258] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.498405 27739 ts_tablet_manager.cc:1875] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.502094 27739 ts_tablet_manager.cc:1888] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.502159 27739 log.cc:1196] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/fc67bd4c105d49c3afe3940637db5ab9
I20240502 01:58:40.502355 27739 ts_tablet_manager.cc:1909] T fc67bd4c105d49c3afe3940637db5ab9 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:40.502492 27740 ts_tablet_manager.cc:1888] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20240502 01:58:40.504212 24965 consensus_peers.cc:485] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): Couldn't send request to peer 588bfc090a5247598cd5d3cee5ad2007. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.504268 27740 log.cc:1196] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a51df555e1143b8b8db1d70f7407a1f
I20240502 01:58:40.504814 27740 ts_tablet_manager.cc:1909] T 7a51df555e1143b8b8db1d70f7407a1f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:40.505518 27740 tablet_replica.cc:330] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:40.505710 27740 raft_consensus.cc:2229] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:40.505687 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 7a51df555e1143b8b8db1d70f7407a1f (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.505816 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet fc67bd4c105d49c3afe3940637db5ab9 (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.506047 27740 raft_consensus.cc:2258] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.506587 27864 tablet_replica.cc:330] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:40.506582 27740 ts_tablet_manager.cc:1875] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.506762 27864 raft_consensus.cc:2229] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.506908 27864 raft_consensus.cc:2258] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.508258 27864 ts_tablet_manager.cc:1875] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.510708 27864 ts_tablet_manager.cc:1888] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.510787 27864 log.cc:1196] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a51df555e1143b8b8db1d70f7407a1f
I20240502 01:58:40.511623 27864 ts_tablet_manager.cc:1909] T 7a51df555e1143b8b8db1d70f7407a1f P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:40.511914 27740 ts_tablet_manager.cc:1888] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.511994 27740 log.cc:1196] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc67bd4c105d49c3afe3940637db5ab9
I20240502 01:58:40.512280 27740 ts_tablet_manager.cc:1909] T fc67bd4c105d49c3afe3940637db5ab9 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:40.513780 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet fc67bd4c105d49c3afe3940637db5ab9 (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.514639 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 7a51df555e1143b8b8db1d70f7407a1f (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.515059 27864 tablet_replica.cc:330] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:40.515203 27864 raft_consensus.cc:2229] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:40.515319 27864 raft_consensus.cc:2258] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:40.515743 27864 ts_tablet_manager.cc:1875] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:40.517510 27864 ts_tablet_manager.cc:1888] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:40.517580 27864 log.cc:1196] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/00616380fd3941159bfd4c2a6bba5761
I20240502 01:58:40.517838 27864 ts_tablet_manager.cc:1909] T 00616380fd3941159bfd4c2a6bba5761 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:40.518383 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 00616380fd3941159bfd4c2a6bba5761 (table rebalance_test_table_2 [id=ada69ce4444a4abc96fca6f0b8b62113]) successfully deleted
I20240502 01:58:40.588984 27865 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.592785 27869 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.614953 27736 raft_consensus.cc:482] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.615057 27736 raft_consensus.cc:504] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.615530 27736 leader_election.cc:288] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:40.616045 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83fb5ee990d493eba6e78eca7e747bc" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:40.616206 27446 raft_consensus.cc:2454] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:40.616578 27246 leader_election.cc:302] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:40.616752 27736 raft_consensus.cc:2781] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.616803 27736 raft_consensus.cc:482] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.616842 27736 raft_consensus.cc:3037] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.617805 27736 raft_consensus.cc:504] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.618168 27736 leader_election.cc:288] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:40.618647 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83fb5ee990d493eba6e78eca7e747bc" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:40.618764 27446 raft_consensus.cc:3037] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.619685 27446 raft_consensus.cc:2454] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:40.620038 27246 leader_election.cc:302] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:40.620157 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83fb5ee990d493eba6e78eca7e747bc" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:40.620162 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83fb5ee990d493eba6e78eca7e747bc" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:40.620276 27736 raft_consensus.cc:2781] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.620294 24632 raft_consensus.cc:2454] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:40.620361 27736 raft_consensus.cc:686] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:40.620496 27736 consensus_queue.cc:237] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:40.622300 24484 catalog_manager.cc:5484] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.646665 27290 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:40.650882 24610 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:58:40.658643 25011 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:58:40.661285 27555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.661712 27424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:40.675936 27157 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.677073 24743 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:58:40.686331 24877 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:58:40.746455 25034 consensus_queue.cc:237] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:40.747928 24630 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.748245 24897 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.748338 26891 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.748741 26891 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.750910 26891 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
W20240502 01:58:40.752457 24966 consensus_peers.cc:485] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce35d9953e294b4986ff9e93e12db6f6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.753448 24897 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.753510 24630 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.757283 24478 catalog_manager.cc:5484] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.766479 25034 consensus_queue.cc:237] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:40.767700 24630 raft_consensus.cc:1261] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.767731 24765 raft_consensus.cc:1261] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.768119 27890 consensus_queue.cc:1035] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.768306 27894 consensus_queue.cc:1035] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.770007 27894 raft_consensus.cc:2932] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.770444 24484 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:40.771951 24630 raft_consensus.cc:2932] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
W20240502 01:58:40.772926 24965 consensus_peers.cc:485] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abfd63f6a0e84f3e855e7c07d3b9cc70. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.772950 24478 catalog_manager.cc:5484] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.775820 24765 raft_consensus.cc:2932] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.780340 24630 raft_consensus.cc:3037] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.780604 27446 raft_consensus.cc:1261] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.781059 24630 raft_consensus.cc:1261] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.781167 27736 consensus_queue.cc:1035] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.781386 27649 consensus_queue.cc:1035] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.783861 27912 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:40.783994 27910 mvcc.cc:205] Tried to move back new op lower bound from 7023063534711480320 to 7023063534062964736. Current Snapshot: MvccSnapshot[applied={T|T < 7023063534711480320}]
I20240502 01:58:40.784235 27912 tablet.cc:1620] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:40.786127 24484 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]). Expected version 1 got 0
I20240502 01:58:40.786365 24632 consensus_queue.cc:237] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:40.787145 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f] alter complete (version 1)
I20240502 01:58:40.787778 25034 raft_consensus.cc:1261] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.788152 24765 raft_consensus.cc:1261] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.788309 27917 consensus_queue.cc:1035] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.788648 27895 consensus_queue.cc:1035] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.789724 24486 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]). Expected version 1 got 0
I20240502 01:58:40.790763 27921 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:40.791541 24478 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]). Expected version 1 got 0
I20240502 01:58:40.791587 27921 tablet.cc:1620] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
W20240502 01:58:40.793012 24567 consensus_peers.cc:485] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe6184a46a0d438da3bf1f3427a8c890. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.793408 24486 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f]). Expected version 1 got 0
I20240502 01:58:40.794507 27648 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.794746 27913 mvcc.cc:205] Tried to move back new op lower bound from 7023063534711480320 to 7023063534062964736. Current Snapshot: MvccSnapshot[applied={T|T < 7023063534711480320}]
I20240502 01:58:40.795269 27895 raft_consensus.cc:2932] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.795383 27914 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:40.796290 27914 tablet.cc:1620] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:40.797302 24765 raft_consensus.cc:2932] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.798393 24486 catalog_manager.cc:5484] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.802297 27648 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.803822 25034 raft_consensus.cc:2932] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.810211 25034 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:40.811461 24765 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.811560 24897 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.811769 27890 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.811936 26891 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.813714 27890 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
W20240502 01:58:40.814931 24963 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26b7501639064035a570b3b4c1cf083f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.815790 24765 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.815790 24897 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.816258 24484 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.825085 24765 consensus_queue.cc:237] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:40.826551 24632 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.826946 24897 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.826975 26978 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.827399 26978 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.828729 27942 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.829088 24632 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.830744 24897 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.831739 24486 catalog_manager.cc:5484] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
W20240502 01:58:40.834091 24701 consensus_peers.cc:485] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70d0c3a7075a4e40a621b73931a3a33b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.840435 25034 consensus_queue.cc:237] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:40.841571 24632 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.841701 24897 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.841967 27890 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.842141 27894 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:40.842360 24965 consensus_peers.cc:485] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0345ae8459544e049edf0ef0bcc5136d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.843916 27894 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.844349 24632 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.844491 24897 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.846776 24484 catalog_manager.cc:5484] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.855028 24897 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:40.856407 25034 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.856686 24765 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.856765 27093 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.857103 27093 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.858894 27960 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.859293 24765 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.860415 25034 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.861851 24485 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
W20240502 01:58:40.862386 24833 consensus_peers.cc:485] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91d0d9525b5f4384861dc50d0bf4967d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.869518 24632 consensus_queue.cc:237] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:40.870936 24765 raft_consensus.cc:1261] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.870971 24897 raft_consensus.cc:1261] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.871775 27917 consensus_queue.cc:1035] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.871809 27736 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.871986 27923 consensus_queue.cc:1035] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.872710 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:40.875049 27747 raft_consensus.cc:2932] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
W20240502 01:58:40.875438 24564 consensus_peers.cc:485] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6846b4da6654802b00203529a01e398. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.876260 24765 raft_consensus.cc:2932] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.879256 24485 catalog_manager.cc:5484] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.879645 24897 raft_consensus.cc:2932] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.880585 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 8c28e5d1164e4dfb997f2d7fec77429e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.880908 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e5d1164e4dfb997f2d7fec77429e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.881887 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 6560bc797c024a90aa87515d0ef2c4c6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.882171 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6560bc797c024a90aa87515d0ef2c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.883038 27973 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:40.883965 27973 tablet_bootstrap.cc:654] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.885238 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 8c28e5d1164e4dfb997f2d7fec77429e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.885495 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e5d1164e4dfb997f2d7fec77429e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.885474 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 6560bc797c024a90aa87515d0ef2c4c6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.885744 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6560bc797c024a90aa87515d0ef2c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.885967 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 8c28e5d1164e4dfb997f2d7fec77429e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.886209 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e5d1164e4dfb997f2d7fec77429e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.886272 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0629c2f9014b5dad64e46658818f29 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.886482 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0629c2f9014b5dad64e46658818f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.887763 27744 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:40.887729 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0629c2f9014b5dad64e46658818f29 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.887943 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0629c2f9014b5dad64e46658818f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.888134 27644 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:40.888612 27744 tablet_bootstrap.cc:654] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.888880 27644 tablet_bootstrap.cc:654] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.888957 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 6560bc797c024a90aa87515d0ef2c4c6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.889194 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6560bc797c024a90aa87515d0ef2c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.890817 27744 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:40.890821 27981 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:40.890903 27744 ts_tablet_manager.cc:1367] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:40.891336 27744 raft_consensus.cc:348] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.891430 27744 raft_consensus.cc:374] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.891464 27744 raft_consensus.cc:729] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.891423 24484 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:40.891582 27744 consensus_queue.cc:260] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.891875 27744 ts_tablet_manager.cc:1398] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.892764 27649 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.892916 27981 tablet_bootstrap.cc:654] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.893822 27981 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:40.893962 27981 ts_tablet_manager.cc:1367] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:40.894333 27981 raft_consensus.cc:348] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.894423 27981 raft_consensus.cc:374] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.894452 27981 raft_consensus.cc:729] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.894557 27981 consensus_queue.cc:260] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.894798 27981 ts_tablet_manager.cc:1398] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.894959 27981 tablet_bootstrap.cc:492] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:40.895411 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0629c2f9014b5dad64e46658818f29 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.895633 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0629c2f9014b5dad64e46658818f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.895671 27981 tablet_bootstrap.cc:654] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.898619 27644 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:40.898701 27644 ts_tablet_manager.cc:1367] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:58:40.899083 27644 raft_consensus.cc:348] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.899176 27644 raft_consensus.cc:374] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.899205 27644 raft_consensus.cc:729] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.899312 27644 consensus_queue.cc:260] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.899566 27644 ts_tablet_manager.cc:1398] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.899724 27644 tablet_bootstrap.cc:492] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:40.900506 27644 tablet_bootstrap.cc:654] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.903450 27975 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:40.905099 27644 tablet_bootstrap.cc:492] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:40.905128 27981 tablet_bootstrap.cc:492] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:40.905186 27644 ts_tablet_manager.cc:1367] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:40.905186 27981 ts_tablet_manager.cc:1367] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 01:58:40.905589 27644 raft_consensus.cc:348] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.905589 27981 raft_consensus.cc:348] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.905692 27981 raft_consensus.cc:374] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.905700 27644 raft_consensus.cc:374] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.905731 27981 raft_consensus.cc:729] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.905731 27644 raft_consensus.cc:729] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.905831 27644 consensus_queue.cc:260] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.905831 27981 consensus_queue.cc:260] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.906088 27644 ts_tablet_manager.cc:1398] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.906268 27981 ts_tablet_manager.cc:1398] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:40.906414 27981 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:40.907002 27975 tablet_bootstrap.cc:654] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.907195 27981 tablet_bootstrap.cc:654] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.908004 27912 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:40.908239 27912 tablet.cc:1620] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:40.908365 27973 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:40.908470 27973 ts_tablet_manager.cc:1367] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240502 01:58:40.908903 27973 raft_consensus.cc:348] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.909003 27973 raft_consensus.cc:374] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.909034 27973 raft_consensus.cc:729] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.909005 27914 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:40.909147 27973 consensus_queue.cc:260] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.909261 27914 tablet.cc:1620] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:40.909394 27973 ts_tablet_manager.cc:1398] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.909556 27973 tablet_bootstrap.cc:492] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:40.910367 27973 tablet_bootstrap.cc:654] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.913318 27921 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:40.913604 27921 tablet.cc:1620] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:40.919070 24478 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=a62a941233ed426696d280c2e6f6444f] alter complete (version 2)
I20240502 01:58:40.920523 27975 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:40.920689 27975 ts_tablet_manager.cc:1367] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:58:40.921087 27973 tablet_bootstrap.cc:492] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:40.921187 27973 ts_tablet_manager.cc:1367] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:58:40.921144 27975 raft_consensus.cc:348] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.921243 27975 raft_consensus.cc:374] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.921267 27975 raft_consensus.cc:729] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.921388 27975 consensus_queue.cc:260] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.921655 27975 ts_tablet_manager.cc:1398] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.921701 27973 raft_consensus.cc:348] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.921808 27973 raft_consensus.cc:374] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.921842 27973 raft_consensus.cc:729] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.922000 27973 consensus_queue.cc:260] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:40.922310 27973 ts_tablet_manager.cc:1398] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.923532 27981 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:40.923638 27981 ts_tablet_manager.cc:1367] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240502 01:58:40.924158 27981 raft_consensus.cc:348] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.924268 27981 raft_consensus.cc:374] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.924300 27981 raft_consensus.cc:729] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.924446 27981 consensus_queue.cc:260] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.924741 27981 ts_tablet_manager.cc:1398] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:40.927460 24632 consensus_queue.cc:237] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:40.928907 25034 raft_consensus.cc:1261] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.929358 27917 consensus_queue.cc:1035] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.930980 24897 raft_consensus.cc:1261] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.931424 27917 consensus_queue.cc:1035] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:40.934681 24567 consensus_peers.cc:485] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aafc084daeec44d5aa2ac9fca5615484. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.936537 27923 raft_consensus.cc:2932] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.938979 25034 raft_consensus.cc:2932] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.939409 24484 catalog_manager.cc:5484] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.945580 24897 raft_consensus.cc:2932] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:40.952594 24897 consensus_queue.cc:237] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:40.954095 25034 raft_consensus.cc:1261] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.954226 24632 raft_consensus.cc:1261] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.954619 27645 consensus_queue.cc:1035] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.955001 27645 consensus_queue.cc:1035] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.956931 27960 raft_consensus.cc:2932] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.957580 24632 raft_consensus.cc:2932] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.959586 25034 raft_consensus.cc:2932] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
W20240502 01:58:40.959903 24832 consensus_peers.cc:485] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 339b2f9649fd4c7c9fbec0999f958e7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.960732 24484 catalog_manager.cc:5484] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:40.966367 27890 raft_consensus.cc:482] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.966460 27890 raft_consensus.cc:504] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.966763 27890 leader_election.cc:288] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:40.967149 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6560bc797c024a90aa87515d0ef2c4c6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:40.967275 27177 raft_consensus.cc:2454] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:40.967517 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6560bc797c024a90aa87515d0ef2c4c6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:40.967588 24965 leader_election.cc:302] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:40.967628 24632 raft_consensus.cc:2454] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:40.967751 27890 raft_consensus.cc:2781] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.967800 27890 raft_consensus.cc:482] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.967834 27890 raft_consensus.cc:3037] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.967864 24897 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:40.968717 27890 raft_consensus.cc:504] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.969051 27890 leader_election.cc:288] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:40.969375 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6560bc797c024a90aa87515d0ef2c4c6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:40.969491 27177 raft_consensus.cc:3037] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.969480 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6560bc797c024a90aa87515d0ef2c4c6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:40.969563 24632 raft_consensus.cc:3037] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.969672 24630 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:40.970057 24833 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3edec06b3d8450a8cc3002be70c848a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.970140 27645 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.970337 25034 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.970383 24632 raft_consensus.cc:2454] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:40.970732 24965 leader_election.cc:302] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:40.970793 27177 raft_consensus.cc:2454] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:40.970796 27960 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.971073 26891 raft_consensus.cc:2781] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.971196 26891 raft_consensus.cc:686] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:40.971323 26891 consensus_queue.cc:237] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:40.972517 27960 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.972908 24632 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.972972 25034 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:40.974668 24486 catalog_manager.cc:5484] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.975032 24479 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.983734 24632 consensus_queue.cc:237] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:40.984982 25034 raft_consensus.cc:1261] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.985200 24765 raft_consensus.cc:1261] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:40.985350 24564 consensus_peers.cc:485] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5340b3277654975bbbdfb4312b7c00d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:40.985435 27923 consensus_queue.cc:1035] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.985677 27747 consensus_queue.cc:1035] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.987413 27923 raft_consensus.cc:2932] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.987958 25034 raft_consensus.cc:2932] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.987983 24765 raft_consensus.cc:2932] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:40.990659 24484 catalog_manager.cc:5484] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:40.996665 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 01:58:40.996766 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 01:58:40.997032 24765 consensus_queue.cc:237] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:40.998548 25034 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.998657 24897 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:40.999174 26978 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:40.999397 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 4844b449d79d425f86b3076f9f43404f
I20240502 01:58:40.999702 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet b5340b3277654975bbbdfb4312b7c00d
I20240502 01:58:40.999934 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 803c34f634d6480a862eeb642aaee3cc
I20240502 01:58:41.000368 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4844b449d79d425f86b3076f9f43404f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.000463 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet b5340b3277654975bbbdfb4312b7c00d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.000598 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet b5340b3277654975bbbdfb4312b7c00d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:41.000864 27739 tablet_replica.cc:330] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.000962 24875 tablet_service.cc:1514] Processing DeleteTablet for tablet 803c34f634d6480a862eeb642aaee3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:41.000998 27739 raft_consensus.cc:2229] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.001108 27739 raft_consensus.cc:2258] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.001179 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet b5340b3277654975bbbdfb4312b7c00d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:41.001312 27864 tablet_replica.cc:330] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:41.001430 27864 raft_consensus.cc:2229] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.001549 27739 ts_tablet_manager.cc:1875] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.001592 27864 raft_consensus.cc:2258] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.002249 27864 ts_tablet_manager.cc:1875] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.002368 28027 tablet_replica.cc:330] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:41.002539 28027 raft_consensus.cc:2229] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 01:58:41.002676 24700 consensus_peers.cc:485] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26422e12ddf64e27b3913c60800e5a56. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.002689 28027 raft_consensus.cc:2258] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.003187 28027 ts_tablet_manager.cc:1875] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.003420 27739 ts_tablet_manager.cc:1888] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.003490 27739 log.cc:1196] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/4844b449d79d425f86b3076f9f43404f
I20240502 01:58:41.003662 27739 ts_tablet_manager.cc:1909] T 4844b449d79d425f86b3076f9f43404f P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.003803 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4844b449d79d425f86b3076f9f43404f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:41.003948 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4844b449d79d425f86b3076f9f43404f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:41.004281 27864 ts_tablet_manager.cc:1888] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.004350 27864 log.cc:1196] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/b5340b3277654975bbbdfb4312b7c00d
I20240502 01:58:41.004452 27739 tablet_replica.cc:330] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.004565 27739 raft_consensus.cc:2229] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.004590 27864 ts_tablet_manager.cc:1909] T b5340b3277654975bbbdfb4312b7c00d P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:41.004635 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 803c34f634d6480a862eeb642aaee3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:41.004782 27740 tablet_replica.cc:330] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:41.004784 27739 raft_consensus.cc:2258] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.004921 27740 raft_consensus.cc:2229] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.005039 27740 raft_consensus.cc:2258] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.005365 27739 ts_tablet_manager.cc:1875] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.005321 27941 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.005465 27740 ts_tablet_manager.cc:1875] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.005868 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 803c34f634d6480a862eeb642aaee3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.006654 28027 ts_tablet_manager.cc:1888] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.006713 28027 log.cc:1196] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/803c34f634d6480a862eeb642aaee3cc
W20240502 01:58:41.007035 24471 catalog_manager.cc:4797] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): delete failed for tablet b5340b3277654975bbbdfb4312b7c00d because the tablet was not found. No further retry: Not found: Tablet not found: b5340b3277654975bbbdfb4312b7c00d
I20240502 01:58:41.007207 28027 ts_tablet_manager.cc:1909] T 803c34f634d6480a862eeb642aaee3cc P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:41.007226 27739 ts_tablet_manager.cc:1888] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.007287 27739 log.cc:1196] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/b5340b3277654975bbbdfb4312b7c00d
I20240502 01:58:41.005875 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 4844b449d79d425f86b3076f9f43404f (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.007474 27740 ts_tablet_manager.cc:1888] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.007514 27739 ts_tablet_manager.cc:1909] T b5340b3277654975bbbdfb4312b7c00d P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.007534 27740 log.cc:1196] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/803c34f634d6480a862eeb642aaee3cc
I20240502 01:58:41.007778 27740 ts_tablet_manager.cc:1909] T 803c34f634d6480a862eeb642aaee3cc P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:41.007843 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 803c34f634d6480a862eeb642aaee3cc (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.008025 28027 tablet_replica.cc:330] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:41.008083 27739 tablet_replica.cc:330] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.008129 28027 raft_consensus.cc:2229] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.008195 27739 raft_consensus.cc:2229] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.008244 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet b5340b3277654975bbbdfb4312b7c00d (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.008284 28027 raft_consensus.cc:2258] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.008291 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet b5340b3277654975bbbdfb4312b7c00d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:40 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:41.008352 27739 raft_consensus.cc:2258] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.008544 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 803c34f634d6480a862eeb642aaee3cc (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.008767 27739 ts_tablet_manager.cc:1875] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.008796 28027 ts_tablet_manager.cc:1875] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.009222 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet b5340b3277654975bbbdfb4312b7c00d (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.009408 27740 tablet_replica.cc:330] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:41.009382 27941 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:41.009701 27740 raft_consensus.cc:2229] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.009788 27864 tablet_replica.cc:330] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:41.009972 27864 raft_consensus.cc:2229] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.010174 27740 raft_consensus.cc:2258] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.010335 27864 raft_consensus.cc:2258] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.010560 28027 ts_tablet_manager.cc:1888] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.010670 28027 log.cc:1196] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/4844b449d79d425f86b3076f9f43404f
I20240502 01:58:41.010941 28027 ts_tablet_manager.cc:1909] T 4844b449d79d425f86b3076f9f43404f P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:41.011008 27740 ts_tablet_manager.cc:1875] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.010998 24897 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:41.011221 27739 ts_tablet_manager.cc:1888] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.011341 27739 log.cc:1196] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/803c34f634d6480a862eeb642aaee3cc
I20240502 01:58:41.011564 27864 ts_tablet_manager.cc:1875] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.011550 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 4844b449d79d425f86b3076f9f43404f (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.011665 27739 ts_tablet_manager.cc:1909] T 803c34f634d6480a862eeb642aaee3cc P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.012765 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 803c34f634d6480a862eeb642aaee3cc (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.013487 24486 catalog_manager.cc:5484] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:41.014739 27864 ts_tablet_manager.cc:1888] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.014815 27864 log.cc:1196] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/4844b449d79d425f86b3076f9f43404f
I20240502 01:58:41.015060 27864 ts_tablet_manager.cc:1909] T 4844b449d79d425f86b3076f9f43404f P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:41.015627 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 4844b449d79d425f86b3076f9f43404f (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.015988 27740 ts_tablet_manager.cc:1888] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.016057 27740 log.cc:1196] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/b5340b3277654975bbbdfb4312b7c00d
I20240502 01:58:41.016238 27740 ts_tablet_manager.cc:1909] T b5340b3277654975bbbdfb4312b7c00d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:41.016716 25032 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:41.017851 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet b5340b3277654975bbbdfb4312b7c00d (table rebalance_test_table_4 [id=dc73c03fa3e04cbdbe5c175300d0a21a]) successfully deleted
I20240502 01:58:41.021001 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:41.022933 25032 consensus_queue.cc:237] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
W20240502 01:58:41.024744 24966 consensus_peers.cc:485] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a7e0e4e54f47849bb29d1e7ae029af. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.024931 24632 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.025226 24897 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.025413 27890 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.025575 26891 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.026753 27912 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:41.027206 26891 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.027640 27921 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:41.027617 24630 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.031054 27914 tablet.cc:1722] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:41.032274 24484 catalog_manager.cc:5484] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.035213 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=a62a941233ed426696d280c2e6f6444f] alter complete (version 3)
I20240502 01:58:41.035315 24897 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.039362 24632 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
W20240502 01:58:41.040594 24567 consensus_peers.cc:485] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69182e5f4ae7401f89d22f063fa8df63. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.040607 25032 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.040740 24897 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.040956 27747 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.041172 27895 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.042721 27917 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.043136 25032 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.043148 24897 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.045982 24478 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.052551 24632 consensus_queue.cc:237] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.053864 24897 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.053895 24765 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:41.054119 24564 consensus_peers.cc:485] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ede6fc65d044804848e6f61e8c238c0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.054316 27747 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.054541 27923 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.055975 27923 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.056353 24765 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.058284 24897 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.058871 24484 catalog_manager.cc:5484] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.067579 24632 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.069140 24897 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.069149 24765 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.069571 27917 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.069774 27747 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.071727 27917 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
W20240502 01:58:41.072904 24566 consensus_peers.cc:485] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e720d8363bb049938798ab0b9aabc28d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.073648 24764 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:41.074290 24897 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:41.075125 27867 raft_consensus.cc:482] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.075234 27867 raft_consensus.cc:504] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.075738 27867 leader_election.cc:288] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:41.075836 24479 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:41.077574 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0629c2f9014b5dad64e46658818f29" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:41.077706 27177 raft_consensus.cc:2454] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:41.078059 27513 leader_election.cc:302] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:41.078239 27867 raft_consensus.cc:2781] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.078296 27867 raft_consensus.cc:482] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.078337 27867 raft_consensus.cc:3037] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.079098 27867 raft_consensus.cc:504] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.079401 27867 leader_election.cc:288] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:41.079721 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0629c2f9014b5dad64e46658818f29" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:41.079826 27177 raft_consensus.cc:3037] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.080716 27177 raft_consensus.cc:2454] T 5b0629c2f9014b5dad64e46658818f29 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:41.081317 27513 leader_election.cc:302] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:41.081327 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0629c2f9014b5dad64e46658818f29" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:41.081459 27312 raft_consensus.cc:2454] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:41.081470 27867 raft_consensus.cc:2781] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.081552 27867 raft_consensus.cc:686] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:41.081669 27867 consensus_queue.cc:237] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.081344 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0629c2f9014b5dad64e46658818f29" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:41.082068 27311 raft_consensus.cc:3037] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.082903 27311 raft_consensus.cc:2454] T 5b0629c2f9014b5dad64e46658818f29 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:41.084177 24484 catalog_manager.cc:5484] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:41.086688 24764 consensus_queue.cc:237] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.087953 24632 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.087953 24897 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.088138 27926 raft_consensus.cc:482] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.088225 27926 raft_consensus.cc:504] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
W20240502 01:58:41.088346 24701 consensus_peers.cc:485] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a383190824e4611a9e279b05bfe4e2a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.088403 27942 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.088578 27926 leader_election.cc:288] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:41.088873 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e5d1164e4dfb997f2d7fec77429e" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:41.088898 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e5d1164e4dfb997f2d7fec77429e" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:41.088920 27941 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.088994 27311 raft_consensus.cc:2454] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:41.089012 27578 raft_consensus.cc:2454] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:41.089238 27110 leader_election.cc:302] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:41.089409 27926 raft_consensus.cc:2781] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.089475 27926 raft_consensus.cc:482] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.089520 27926 raft_consensus.cc:3037] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.090445 27926 raft_consensus.cc:504] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.090533 27942 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.090766 27926 leader_election.cc:288] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:41.091178 24632 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.091246 24897 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.091794 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e5d1164e4dfb997f2d7fec77429e" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:41.091884 27311 raft_consensus.cc:3037] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.092743 27311 raft_consensus.cc:2454] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:41.093106 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e5d1164e4dfb997f2d7fec77429e" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:41.093206 27578 raft_consensus.cc:3037] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.094048 27578 raft_consensus.cc:2454] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:41.094295 27113 leader_election.cc:302] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:41.094261 24479 catalog_manager.cc:5484] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:41.094771 27926 raft_consensus.cc:2781] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.094854 27926 raft_consensus.cc:686] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:41.094974 27926 consensus_queue.cc:237] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.102418 24486 catalog_manager.cc:5484] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:41.103981 24632 consensus_queue.cc:237] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.105340 25032 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.105367 24897 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:41.105660 24567 consensus_peers.cc:485] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 752d360cfdd64846b6cd37c9b859d7a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.105738 27747 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.106060 27747 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.107795 27923 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.109472 25034 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.110309 24486 catalog_manager.cc:5484] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:41.111747 24897 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:41.123528 24632 consensus_queue.cc:237] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.124711 24764 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.124917 25034 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:41.124933 24564 consensus_peers.cc:485] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a094659d539e4c4f9861e356168c5894. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.125058 27895 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.125319 27923 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.126698 27895 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.127290 24765 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.127538 25034 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.129791 24484 catalog_manager.cc:5484] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:41.139178 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:41.139298 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:41.140925 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet f83fb5ee990d493eba6e78eca7e747bc
I20240502 01:58:41.141525 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet f83fb5ee990d493eba6e78eca7e747bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:41.141535 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet f83fb5ee990d493eba6e78eca7e747bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.141507 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet f83fb5ee990d493eba6e78eca7e747bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:41.141676 28026 tablet_replica.cc:330] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:41.141775 28026 raft_consensus.cc:2229] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.141968 27739 tablet_replica.cc:330] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.141973 28026 raft_consensus.cc:2258] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.142024 28099 tablet_replica.cc:330] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:41.142084 27739 raft_consensus.cc:2229] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.142181 27739 raft_consensus.cc:2258] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.142333 28099 raft_consensus.cc:2229] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.142578 28099 raft_consensus.cc:2258] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.142995 27739 ts_tablet_manager.cc:1875] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.143807 28099 ts_tablet_manager.cc:1875] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.144666 27739 ts_tablet_manager.cc:1888] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.144724 27739 log.cc:1196] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/f83fb5ee990d493eba6e78eca7e747bc
I20240502 01:58:41.144919 27739 ts_tablet_manager.cc:1909] T f83fb5ee990d493eba6e78eca7e747bc P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.145718 28026 ts_tablet_manager.cc:1875] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.145890 28099 ts_tablet_manager.cc:1888] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.146054 28099 log.cc:1196] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/f83fb5ee990d493eba6e78eca7e747bc
I20240502 01:58:41.146428 28099 ts_tablet_manager.cc:1909] T f83fb5ee990d493eba6e78eca7e747bc P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:41.147760 28026 ts_tablet_manager.cc:1888] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.147836 28026 log.cc:1196] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/f83fb5ee990d493eba6e78eca7e747bc
I20240502 01:58:41.147995 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table_new_name [id=a62a941233ed426696d280c2e6f6444f]) successfully deleted
I20240502 01:58:41.148084 28026 ts_tablet_manager.cc:1909] T f83fb5ee990d493eba6e78eca7e747bc P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:41.148715 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table_new_name [id=a62a941233ed426696d280c2e6f6444f]) successfully deleted
I20240502 01:58:41.149085 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet f83fb5ee990d493eba6e78eca7e747bc (table rebalancer_dynamic_table_new_name [id=a62a941233ed426696d280c2e6f6444f]) successfully deleted
I20240502 01:58:41.151187 28105 ts_tablet_manager.cc:902] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:41.151887 28105 tablet_copy_client.cc:308] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:41.155689 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet ce35d9953e294b4986ff9e93e12db6f6 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:45689)
I20240502 01:58:41.155807 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet ce35d9953e294b4986ff9e93e12db6f6 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:45689: session id = bb653087e8554c7591cf547dd969f3be-ce35d9953e294b4986ff9e93e12db6f6
I20240502 01:58:41.157086 25054 tablet_copy_source_session.cc:205] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.158668 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce35d9953e294b4986ff9e93e12db6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.160696 28105 tablet_copy_client.cc:783] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.160864 28105 tablet_copy_client.cc:647] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.162501 28105 tablet_copy_client.cc:523] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.163872 28105 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:41.181309 28137 ts_tablet_manager.cc:902] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.182076 28137 tablet_copy_client.cc:308] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.188438 28105 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.188817 28105 tablet_bootstrap.cc:492] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:41.189276 28105 ts_tablet_manager.cc:1367] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.025s user 0.026s sys 0.000s
I20240502 01:58:41.189751 28105 raft_consensus.cc:348] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.189888 28105 raft_consensus.cc:729] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:41.190017 28105 consensus_queue.cc:260] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.191228 28105 ts_tablet_manager.cc:1398] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:41.191884 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-ce35d9953e294b4986ff9e93e12db6f6 received from {username='slave'} at 127.17.243.8:45689
I20240502 01:58:41.191884 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet fe6184a46a0d438da3bf1f3427a8c890 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:51949)
I20240502 01:58:41.191980 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session bb653087e8554c7591cf547dd969f3be-ce35d9953e294b4986ff9e93e12db6f6 on tablet ce35d9953e294b4986ff9e93e12db6f6 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:41.192013 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet fe6184a46a0d438da3bf1f3427a8c890 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:51949: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-fe6184a46a0d438da3bf1f3427a8c890
I20240502 01:58:41.194229 24652 tablet_copy_source_session.cc:205] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.195660 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6184a46a0d438da3bf1f3427a8c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.197912 28137 tablet_copy_client.cc:783] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.198133 28137 tablet_copy_client.cc:647] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.200289 28137 tablet_copy_client.cc:523] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.201650 28137 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.225095 28165 ts_tablet_manager.cc:902] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:41.225901 28165 tablet_copy_client.cc:308] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:41.229466 28137 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.229748 28137 tablet_bootstrap.cc:492] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:41.229952 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:54405)
I20240502 01:58:41.230046 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet abfd63f6a0e84f3e855e7c07d3b9cc70 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:54405: session id = 9dd619a595e94004ac0f7f5a9054e7cc-abfd63f6a0e84f3e855e7c07d3b9cc70
I20240502 01:58:41.230867 25054 tablet_copy_source_session.cc:205] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.231235 28137 ts_tablet_manager.cc:1367] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.004s
I20240502 01:58:41.231722 28137 raft_consensus.cc:348] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.231868 28137 raft_consensus.cc:729] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:41.231974 28137 consensus_queue.cc:260] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.232254 28137 ts_tablet_manager.cc:1398] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.232808 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-fe6184a46a0d438da3bf1f3427a8c890 received from {username='slave'} at 127.17.243.7:51949
I20240502 01:58:41.232898 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-fe6184a46a0d438da3bf1f3427a8c890 on tablet fe6184a46a0d438da3bf1f3427a8c890 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:41.232993 28165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfd63f6a0e84f3e855e7c07d3b9cc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.235329 28165 tablet_copy_client.cc:783] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.235459 28165 tablet_copy_client.cc:647] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.236382 28105 ts_tablet_manager.cc:902] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:41.236805 28165 tablet_copy_client.cc:523] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.236821 28105 tablet_copy_client.cc:308] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:41.238289 28165 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.241709 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 70d0c3a7075a4e40a621b73931a3a33b from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:39965)
I20240502 01:58:41.241839 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 70d0c3a7075a4e40a621b73931a3a33b from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:39965: session id = bb653087e8554c7591cf547dd969f3be-70d0c3a7075a4e40a621b73931a3a33b
I20240502 01:58:41.243044 24785 tablet_copy_source_session.cc:205] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.243247 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:41.244544 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d0c3a7075a4e40a621b73931a3a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.246632 28105 tablet_copy_client.cc:783] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.246781 28105 tablet_copy_client.cc:647] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.248361 28105 tablet_copy_client.cc:523] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.249778 28105 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:41.252254 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]), partition=
I20240502 01:58:41.252254 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]), partition=
I20240502 01:58:41.252512 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.252512 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.254884 28197 tablet_bootstrap.cc:492] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.255192 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]), partition=
I20240502 01:58:41.255484 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.255990 28197 tablet_bootstrap.cc:654] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.257071 28197 tablet_bootstrap.cc:492] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:41.257153 28197 ts_tablet_manager.cc:1367] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:41.257339 27644 tablet_bootstrap.cc:492] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.257583 28197 raft_consensus.cc:348] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.257719 28197 raft_consensus.cc:374] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.257802 28197 raft_consensus.cc:729] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.258242 27644 tablet_bootstrap.cc:654] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.258226 28197 consensus_queue.cc:260] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.261369 28197 ts_tablet_manager.cc:1398] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:41.263259 27644 tablet_bootstrap.cc:492] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:41.263357 27644 ts_tablet_manager.cc:1367] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:41.263789 27644 raft_consensus.cc:348] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.263904 27644 raft_consensus.cc:374] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.263942 27644 raft_consensus.cc:729] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.264061 27644 consensus_queue.cc:260] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.264315 27644 ts_tablet_manager.cc:1398] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.274139 28198 tablet_bootstrap.cc:492] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:41.274657 28105 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.274880 28105 tablet_bootstrap.cc:492] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:41.274948 28198 tablet_bootstrap.cc:654] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.275043 28105 ts_tablet_manager.cc:1367] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240502 01:58:41.275403 28105 raft_consensus.cc:348] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.275533 28105 raft_consensus.cc:729] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:41.275637 28105 consensus_queue.cc:260] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.275915 28105 ts_tablet_manager.cc:1398] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.276487 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-70d0c3a7075a4e40a621b73931a3a33b received from {username='slave'} at 127.17.243.8:39965
I20240502 01:58:41.276585 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session bb653087e8554c7591cf547dd969f3be-70d0c3a7075a4e40a621b73931a3a33b on tablet 70d0c3a7075a4e40a621b73931a3a33b with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:41.278434 28198 tablet_bootstrap.cc:492] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:41.278581 28198 ts_tablet_manager.cc:1367] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:41.279045 28198 raft_consensus.cc:348] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.279140 28198 raft_consensus.cc:374] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.279170 28198 raft_consensus.cc:729] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.279281 28198 consensus_queue.cc:260] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.279554 28198 ts_tablet_manager.cc:1398] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.284426 28165 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.284734 28165 tablet_bootstrap.cc:492] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:41.284922 28165 ts_tablet_manager.cc:1367] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.002s
I20240502 01:58:41.285337 28165 raft_consensus.cc:348] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.285461 28165 raft_consensus.cc:729] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:41.285557 28165 consensus_queue.cc:260] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.285806 28165 ts_tablet_manager.cc:1398] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.286509 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-abfd63f6a0e84f3e855e7c07d3b9cc70 received from {username='slave'} at 127.17.243.5:54405
I20240502 01:58:41.286579 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-abfd63f6a0e84f3e855e7c07d3b9cc70 on tablet abfd63f6a0e84f3e855e7c07d3b9cc70 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:41.329998 28165 ts_tablet_manager.cc:902] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:41.330428 28165 tablet_copy_client.cc:308] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:41.335711 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 339b2f9649fd4c7c9fbec0999f958e7f from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:42087)
I20240502 01:58:41.335844 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 339b2f9649fd4c7c9fbec0999f958e7f from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:42087: session id = 9dd619a595e94004ac0f7f5a9054e7cc-339b2f9649fd4c7c9fbec0999f958e7f
I20240502 01:58:41.337271 24918 tablet_copy_source_session.cc:205] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.338605 28165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b2f9649fd4c7c9fbec0999f958e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.340286 28165 tablet_copy_client.cc:783] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.340438 28165 tablet_copy_client.cc:647] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.341917 28165 tablet_copy_client.cc:523] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.344636 28165 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.357673 28137 ts_tablet_manager.cc:902] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:41.358170 28137 tablet_copy_client.cc:308] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:41.359879 28260 ts_tablet_manager.cc:902] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:41.360621 28260 tablet_copy_client.cc:308] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:41.361070 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 0345ae8459544e049edf0ef0bcc5136d from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:54405)
I20240502 01:58:41.361162 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 0345ae8459544e049edf0ef0bcc5136d from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:54405: session id = 9dd619a595e94004ac0f7f5a9054e7cc-0345ae8459544e049edf0ef0bcc5136d
I20240502 01:58:41.362037 25054 tablet_copy_source_session.cc:205] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.362852 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet b3edec06b3d8450a8cc3002be70c848a from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:55633)
I20240502 01:58:41.362939 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet b3edec06b3d8450a8cc3002be70c848a from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:55633: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-b3edec06b3d8450a8cc3002be70c848a
I20240502 01:58:41.363770 24918 tablet_copy_source_session.cc:205] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.364527 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3edec06b3d8450a8cc3002be70c848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.364531 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0345ae8459544e049edf0ef0bcc5136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.366621 28137 tablet_copy_client.cc:783] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.366768 28137 tablet_copy_client.cc:647] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.368752 27017 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.370821 28165 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.371062 28165 tablet_bootstrap.cc:492] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:41.371243 28165 ts_tablet_manager.cc:1367] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.001s
I20240502 01:58:41.371676 28165 raft_consensus.cc:348] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.371779 28165 raft_consensus.cc:729] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:41.371882 28165 consensus_queue.cc:260] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.372506 28260 tablet_copy_client.cc:783] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.372682 28260 tablet_copy_client.cc:647] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.372833 28165 ts_tablet_manager.cc:1398] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:41.373435 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-339b2f9649fd4c7c9fbec0999f958e7f received from {username='slave'} at 127.17.243.5:42087
I20240502 01:58:41.373519 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-339b2f9649fd4c7c9fbec0999f958e7f on tablet 339b2f9649fd4c7c9fbec0999f958e7f with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:41.374433 28260 tablet_copy_client.cc:523] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.375560 28260 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.377532 27894 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.380368 28137 tablet_copy_client.cc:523] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.381453 28137 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.390158 28274 ts_tablet_manager.cc:902] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:41.391340 28274 tablet_copy_client.cc:308] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:41.401027 28260 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.401280 28260 tablet_bootstrap.cc:492] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:41.401425 28260 ts_tablet_manager.cc:1367] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.000s
I20240502 01:58:41.401818 28260 raft_consensus.cc:348] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.401964 28260 raft_consensus.cc:729] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:41.402060 28260 consensus_queue.cc:260] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.402402 28260 ts_tablet_manager.cc:1398] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:58:41.402794 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-0345ae8459544e049edf0ef0bcc5136d received from {username='slave'} at 127.17.243.5:54405
I20240502 01:58:41.402868 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-0345ae8459544e049edf0ef0bcc5136d on tablet 0345ae8459544e049edf0ef0bcc5136d with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:41.406157 28137 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.406420 28137 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:41.406426 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 26b7501639064035a570b3b4c1cf083f from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:43875)
I20240502 01:58:41.406517 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 26b7501639064035a570b3b4c1cf083f from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:43875: session id = 1049b4c8783b48f39be22bd113ad349f-26b7501639064035a570b3b4c1cf083f
I20240502 01:58:41.406742 28137 ts_tablet_manager.cc:1367] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240502 01:58:41.407111 28137 raft_consensus.cc:348] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.407205 28137 raft_consensus.cc:729] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:41.407258 25054 tablet_copy_source_session.cc:205] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.407300 28137 consensus_queue.cc:260] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.407600 28137 ts_tablet_manager.cc:1398] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.408206 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-b3edec06b3d8450a8cc3002be70c848a received from {username='slave'} at 127.17.243.7:55633
I20240502 01:58:41.408291 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-b3edec06b3d8450a8cc3002be70c848a on tablet b3edec06b3d8450a8cc3002be70c848a with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:41.409057 28274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7501639064035a570b3b4c1cf083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.409235 28105 ts_tablet_manager.cc:902] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:41.409639 28105 tablet_copy_client.cc:308] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:41.410248 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 61a7e0e4e54f47849bb29d1e7ae029af from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:45689)
I20240502 01:58:41.410326 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 61a7e0e4e54f47849bb29d1e7ae029af from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:45689: session id = bb653087e8554c7591cf547dd969f3be-61a7e0e4e54f47849bb29d1e7ae029af
I20240502 01:58:41.411083 25054 tablet_copy_source_session.cc:205] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.411660 28274 tablet_copy_client.cc:783] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.411878 28274 tablet_copy_client.cc:647] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.411913 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a7e0e4e54f47849bb29d1e7ae029af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.413367 28274 tablet_copy_client.cc:523] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.413656 28105 tablet_copy_client.cc:783] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.413800 28105 tablet_copy_client.cc:647] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.414932 28274 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.418547 28105 tablet_copy_client.cc:523] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.419657 28105 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:41.422257 28288 ts_tablet_manager.cc:902] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.422822 28288 tablet_copy_client.cc:308] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.426424 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet a6846b4da6654802b00203529a01e398 from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:59275)
I20240502 01:58:41.426533 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet a6846b4da6654802b00203529a01e398 from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:59275: session id = 1049b4c8783b48f39be22bd113ad349f-a6846b4da6654802b00203529a01e398
I20240502 01:58:41.427430 24652 tablet_copy_source_session.cc:205] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.428196 28288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6846b4da6654802b00203529a01e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.429522 28137 ts_tablet_manager.cc:902] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:41.429919 28137 tablet_copy_client.cc:308] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:41.430339 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 91d0d9525b5f4384861dc50d0bf4967d from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:55633)
I20240502 01:58:41.430454 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 91d0d9525b5f4384861dc50d0bf4967d from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:55633: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-91d0d9525b5f4384861dc50d0bf4967d
I20240502 01:58:41.431326 24918 tablet_copy_source_session.cc:205] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.431888 28288 tablet_copy_client.cc:783] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.432101 28288 tablet_copy_client.cc:647] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.432128 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0d9525b5f4384861dc50d0bf4967d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.433594 28288 tablet_copy_client.cc:523] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.434074 28137 tablet_copy_client.cc:783] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.434212 28137 tablet_copy_client.cc:647] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.435088 28288 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.443046 28137 tablet_copy_client.cc:523] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.444162 28137 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.447877 28274 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.448156 28274 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:41.448171 28105 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.448325 28274 ts_tablet_manager.cc:1367] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.002s
I20240502 01:58:41.448385 28105 tablet_bootstrap.cc:492] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:41.448580 28105 ts_tablet_manager.cc:1367] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.005s
I20240502 01:58:41.448875 28274 raft_consensus.cc:348] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.449012 28274 raft_consensus.cc:729] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:41.449021 28105 raft_consensus.cc:348] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.449137 28105 raft_consensus.cc:729] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:41.449119 28274 consensus_queue.cc:260] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.449224 28105 consensus_queue.cc:260] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.449469 28105 ts_tablet_manager.cc:1398] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.450021 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-61a7e0e4e54f47849bb29d1e7ae029af received from {username='slave'} at 127.17.243.8:45689
I20240502 01:58:41.450098 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session bb653087e8554c7591cf547dd969f3be-61a7e0e4e54f47849bb29d1e7ae029af on tablet 61a7e0e4e54f47849bb29d1e7ae029af with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:41.451887 28274 ts_tablet_manager.cc:1398] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:41.454447 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-26b7501639064035a570b3b4c1cf083f received from {username='slave'} at 127.17.243.6:43875
I20240502 01:58:41.454551 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-26b7501639064035a570b3b4c1cf083f on tablet 26b7501639064035a570b3b4c1cf083f with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:41.464339 28288 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.464668 28288 tablet_bootstrap.cc:492] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:41.464874 28288 ts_tablet_manager.cc:1367] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.005s
I20240502 01:58:41.465361 28288 raft_consensus.cc:348] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.465502 28288 raft_consensus.cc:729] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:41.465646 28288 consensus_queue.cc:260] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.465919 28288 ts_tablet_manager.cc:1398] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 01:58:41.466516 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-a6846b4da6654802b00203529a01e398 received from {username='slave'} at 127.17.243.6:59275
I20240502 01:58:41.466614 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-a6846b4da6654802b00203529a01e398 on tablet a6846b4da6654802b00203529a01e398 with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:41.474386 28137 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.474723 28137 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:41.474892 28137 ts_tablet_manager.cc:1367] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.000s
I20240502 01:58:41.475322 28137 raft_consensus.cc:348] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.475440 28137 raft_consensus.cc:729] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:41.475544 28137 consensus_queue.cc:260] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.475786 28137 ts_tablet_manager.cc:1398] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.476214 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-91d0d9525b5f4384861dc50d0bf4967d received from {username='slave'} at 127.17.243.7:55633
I20240502 01:58:41.476294 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-91d0d9525b5f4384861dc50d0bf4967d on tablet 91d0d9525b5f4384861dc50d0bf4967d with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:41.504768 24485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 01:58:41.504912 24485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 01:58:41.507159 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet cf7bfd4c87a54b4fb31059ed03f955a2
I20240502 01:58:41.507465 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet 339b2f9649fd4c7c9fbec0999f958e7f
I20240502 01:58:41.507650 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7bfd4c87a54b4fb31059ed03f955a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.507743 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7bfd4c87a54b4fb31059ed03f955a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:41.507745 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7bfd4c87a54b4fb31059ed03f955a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:41.507858 27739 tablet_replica.cc:330] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.507936 27740 tablet_replica.cc:330] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:41.507958 27864 tablet_replica.cc:330] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:41.508016 27739 raft_consensus.cc:2229] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.508072 27740 raft_consensus.cc:2229] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.508096 27864 raft_consensus.cc:2229] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.508167 27739 raft_consensus.cc:2258] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.508208 27740 raft_consensus.cc:2258] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.508265 27864 raft_consensus.cc:2258] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.508615 27739 ts_tablet_manager.cc:1875] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.508718 27740 ts_tablet_manager.cc:1875] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.508718 27864 ts_tablet_manager.cc:1875] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.510385 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:41.510459 27739 ts_tablet_manager.cc:1888] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.510516 27739 log.cc:1196] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf7bfd4c87a54b4fb31059ed03f955a2
I20240502 01:58:41.510696 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet fe6184a46a0d438da3bf1f3427a8c890
I20240502 01:58:41.510775 27739 ts_tablet_manager.cc:1909] T cf7bfd4c87a54b4fb31059ed03f955a2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.510767 27740 ts_tablet_manager.cc:1888] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.510767 27864 ts_tablet_manager.cc:1888] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:41.510829 27740 log.cc:1196] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf7bfd4c87a54b4fb31059ed03f955a2
I20240502 01:58:41.510829 27864 log.cc:1196] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/cf7bfd4c87a54b4fb31059ed03f955a2
I20240502 01:58:41.511066 27864 ts_tablet_manager.cc:1909] T cf7bfd4c87a54b4fb31059ed03f955a2 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:41.511066 27740 ts_tablet_manager.cc:1909] T cf7bfd4c87a54b4fb31059ed03f955a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:41.511325 28027 tablet_replica.cc:330] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:41.511448 28027 raft_consensus.cc:2229] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.511488 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:41.511713 28027 raft_consensus.cc:2258] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.511873 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:41.511931 28099 tablet_replica.cc:330] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:41.512050 28099 raft_consensus.cc:2229] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:41.512142 28099 raft_consensus.cc:2258] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:41.512132 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet cf7bfd4c87a54b4fb31059ed03f955a2 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.512262 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.512392 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:41.512465 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet cf7bfd4c87a54b4fb31059ed03f955a2 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.512545 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:41.512631 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6184a46a0d438da3bf1f3427a8c890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:41.512642 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b2f9649fd4c7c9fbec0999f958e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:41 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:41.512804 27740 tablet_replica.cc:330] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:41.512909 27740 raft_consensus.cc:2229] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.513048 27740 raft_consensus.cc:2258] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.513180 27739 tablet_replica.cc:330] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.513298 27739 raft_consensus.cc:2229] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.513430 27739 raft_consensus.cc:2258] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.513446 28099 ts_tablet_manager.cc:1875] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.513651 27864 tablet_replica.cc:330] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:41.513767 27864 raft_consensus.cc:2229] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.513799 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet cf7bfd4c87a54b4fb31059ed03f955a2 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.513819 27740 ts_tablet_manager.cc:1875] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.513891 27864 raft_consensus.cc:2258] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.514515 27739 ts_tablet_manager.cc:1875] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.515221 28099 ts_tablet_manager.cc:1888] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.515285 28099 log.cc:1196] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/fe6184a46a0d438da3bf1f3427a8c890
I20240502 01:58:41.515532 28099 ts_tablet_manager.cc:1909] T fe6184a46a0d438da3bf1f3427a8c890 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:41.515604 27740 ts_tablet_manager.cc:1888] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.515662 27740 log.cc:1196] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/fe6184a46a0d438da3bf1f3427a8c890
I20240502 01:58:41.515934 27740 ts_tablet_manager.cc:1909] T fe6184a46a0d438da3bf1f3427a8c890 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:41.516177 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet fe6184a46a0d438da3bf1f3427a8c890 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.516486 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet fe6184a46a0d438da3bf1f3427a8c890 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.516533 27740 tablet_replica.cc:330] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:41.516459 27867 consensus_queue.cc:1035] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.516587 28027 ts_tablet_manager.cc:1875] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.516631 27740 raft_consensus.cc:2229] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:41.516732 27740 raft_consensus.cc:2258] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.517155 27864 ts_tablet_manager.cc:1875] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.517299 27740 ts_tablet_manager.cc:1875] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.518579 28027 ts_tablet_manager.cc:1888] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.518889 27740 ts_tablet_manager.cc:1888] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.518954 27740 log.cc:1196] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/339b2f9649fd4c7c9fbec0999f958e7f
I20240502 01:58:41.519223 27740 ts_tablet_manager.cc:1909] T 339b2f9649fd4c7c9fbec0999f958e7f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:41.519785 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 339b2f9649fd4c7c9fbec0999f958e7f (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.519922 28105 ts_tablet_manager.cc:902] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:41.520026 27864 ts_tablet_manager.cc:1888] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.520092 27864 log.cc:1196] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe6184a46a0d438da3bf1f3427a8c890
I20240502 01:58:41.520318 27864 ts_tablet_manager.cc:1909] T fe6184a46a0d438da3bf1f3427a8c890 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:41.520406 28260 ts_tablet_manager.cc:902] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.520385 28105 tablet_copy_client.cc:308] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:41.520841 28027 log.cc:1196] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/339b2f9649fd4c7c9fbec0999f958e7f
I20240502 01:58:41.520838 28260 tablet_copy_client.cc:308] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.521104 28027 ts_tablet_manager.cc:1909] T 339b2f9649fd4c7c9fbec0999f958e7f P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:41.521328 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 1a383190824e4611a9e279b05bfe4e2a from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:39965)
I20240502 01:58:41.521411 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 1a383190824e4611a9e279b05bfe4e2a from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:39965: session id = bb653087e8554c7591cf547dd969f3be-1a383190824e4611a9e279b05bfe4e2a
I20240502 01:58:41.521476 27739 ts_tablet_manager.cc:1888] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.521543 27739 log.cc:1196] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/339b2f9649fd4c7c9fbec0999f958e7f
I20240502 01:58:41.521703 27867 consensus_queue.cc:1035] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.521766 27739 ts_tablet_manager.cc:1909] T 339b2f9649fd4c7c9fbec0999f958e7f P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.522297 27739 tablet_replica.cc:330] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:41.522291 24785 tablet_copy_source_session.cc:205] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.522361 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 339b2f9649fd4c7c9fbec0999f958e7f (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.522408 27739 raft_consensus.cc:2229] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:41.522583 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet e720d8363bb049938798ab0b9aabc28d from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:52035)
I20240502 01:58:41.522660 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet e720d8363bb049938798ab0b9aabc28d from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:52035: session id = 9dd619a595e94004ac0f7f5a9054e7cc-e720d8363bb049938798ab0b9aabc28d
I20240502 01:58:41.522773 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet fe6184a46a0d438da3bf1f3427a8c890 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.523005 27739 raft_consensus.cc:2258] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:41.523515 27739 ts_tablet_manager.cc:1875] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.524367 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a383190824e4611a9e279b05bfe4e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.525104 27739 ts_tablet_manager.cc:1888] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.525156 27739 log.cc:1196] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe6184a46a0d438da3bf1f3427a8c890
I20240502 01:58:41.525321 27739 ts_tablet_manager.cc:1909] T fe6184a46a0d438da3bf1f3427a8c890 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:41.525996 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet fe6184a46a0d438da3bf1f3427a8c890 (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.526489 28105 tablet_copy_client.cc:783] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.526625 28105 tablet_copy_client.cc:647] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.523515 24652 tablet_copy_source_session.cc:205] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.527619 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e720d8363bb049938798ab0b9aabc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.527639 28105 tablet_copy_client.cc:523] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.528360 28300 tablet_replica.cc:330] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:41.528579 28300 raft_consensus.cc:2229] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:41.528738 28300 raft_consensus.cc:2258] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:41.528944 28105 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:41.529489 28300 ts_tablet_manager.cc:1875] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:41.529575 28260 tablet_copy_client.cc:783] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.529722 28260 tablet_copy_client.cc:647] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.530083 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 339b2f9649fd4c7c9fbec0999f958e7f (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.531275 28300 ts_tablet_manager.cc:1888] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:41.531710 28300 log.cc:1196] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/339b2f9649fd4c7c9fbec0999f958e7f
I20240502 01:58:41.532025 28300 ts_tablet_manager.cc:1909] T 339b2f9649fd4c7c9fbec0999f958e7f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:41.532819 28260 tablet_copy_client.cc:523] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.533313 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 339b2f9649fd4c7c9fbec0999f958e7f (table rebalance_test_table_6 [id=2c2f8d6ccf20422c9dc000e58437daed]) successfully deleted
I20240502 01:58:41.533735 28260 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.537438 27930 raft_consensus.cc:482] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.537528 27930 raft_consensus.cc:504] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.538506 27930 leader_election.cc:288] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:41.538763 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa2bc99e2ba4fc5998f6af3e5ac47c7" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:41.538879 24896 raft_consensus.cc:2454] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:41.539186 27380 leader_election.cc:302] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:41.539407 27930 raft_consensus.cc:2781] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.539449 27930 raft_consensus.cc:482] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.539479 27930 raft_consensus.cc:3037] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.540161 27930 raft_consensus.cc:504] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.540472 27930 leader_election.cc:288] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:41.540855 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa2bc99e2ba4fc5998f6af3e5ac47c7" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:41.540952 24896 raft_consensus.cc:3037] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.541843 24896 raft_consensus.cc:2454] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:41.542181 27380 leader_election.cc:302] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:41.542248 28165 ts_tablet_manager.cc:902] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:41.542379 27930 raft_consensus.cc:2781] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.542521 27930 raft_consensus.cc:686] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:41.542697 28165 tablet_copy_client.cc:308] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:41.542737 27930 consensus_queue.cc:237] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:41.544224 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa2bc99e2ba4fc5998f6af3e5ac47c7" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:41.544255 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa2bc99e2ba4fc5998f6af3e5ac47c7" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:41.544344 27311 raft_consensus.cc:2454] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:41.546294 24485 catalog_manager.cc:5484] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:41.547451 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 26422e12ddf64e27b3913c60800e5a56 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:47383)
I20240502 01:58:41.547544 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 26422e12ddf64e27b3913c60800e5a56 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:47383: session id = 9dd619a595e94004ac0f7f5a9054e7cc-26422e12ddf64e27b3913c60800e5a56
I20240502 01:58:41.548444 24785 tablet_copy_source_session.cc:205] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.549216 28165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26422e12ddf64e27b3913c60800e5a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.551048 28165 tablet_copy_client.cc:783] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.551198 28165 tablet_copy_client.cc:647] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.552299 28165 tablet_copy_client.cc:523] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.553205 28165 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.554935 28105 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.555176 28105 tablet_bootstrap.cc:492] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:41.555321 28105 ts_tablet_manager.cc:1367] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240502 01:58:41.555737 28105 raft_consensus.cc:348] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.555850 28105 raft_consensus.cc:729] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:41.555946 28105 consensus_queue.cc:260] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.556293 28105 ts_tablet_manager.cc:1398] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.557595 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-1a383190824e4611a9e279b05bfe4e2a received from {username='slave'} at 127.17.243.8:39965
I20240502 01:58:41.557668 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session bb653087e8554c7591cf547dd969f3be-1a383190824e4611a9e279b05bfe4e2a on tablet 1a383190824e4611a9e279b05bfe4e2a with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:41.560329 28105 ts_tablet_manager.cc:902] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.560737 28105 tablet_copy_client.cc:308] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.561743 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet aafc084daeec44d5aa2ac9fca5615484 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:44271)
I20240502 01:58:41.561848 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet aafc084daeec44d5aa2ac9fca5615484 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:44271: session id = bb653087e8554c7591cf547dd969f3be-aafc084daeec44d5aa2ac9fca5615484
I20240502 01:58:41.562793 24652 tablet_copy_source_session.cc:205] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.564026 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafc084daeec44d5aa2ac9fca5615484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.564131 28288 ts_tablet_manager.cc:902] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.564543 28288 tablet_copy_client.cc:308] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.565797 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet a094659d539e4c4f9861e356168c5894 from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:59275)
I20240502 01:58:41.565898 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet a094659d539e4c4f9861e356168c5894 from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:59275: session id = 1049b4c8783b48f39be22bd113ad349f-a094659d539e4c4f9861e356168c5894
I20240502 01:58:41.566138 28105 tablet_copy_client.cc:783] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.566267 28105 tablet_copy_client.cc:647] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.566707 24652 tablet_copy_source_session.cc:205] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.567780 28288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a094659d539e4c4f9861e356168c5894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.569998 28288 tablet_copy_client.cc:783] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.570140 28288 tablet_copy_client.cc:647] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.571427 24743 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:58:41.572959 27424 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:41.579942 27289 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:41.581676 24610 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:58:41.582124 27155 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:41.588217 28158 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.595759 28260 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.596087 28260 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:41.596532 28260 ts_tablet_manager.cc:1367] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.063s user 0.025s sys 0.000s
I20240502 01:58:41.599417 24877 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:58:41.600497 28260 raft_consensus.cc:348] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.600644 28260 raft_consensus.cc:729] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:41.600759 28260 consensus_queue.cc:260] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.601078 28260 ts_tablet_manager.cc:1398] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:58:41.602557 25011 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:58:41.603075 28105 tablet_copy_client.cc:523] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.604419 28105 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:41.607434 28137 ts_tablet_manager.cc:902] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.607875 28137 tablet_copy_client.cc:308] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.608294 24651 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 69182e5f4ae7401f89d22f063fa8df63 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:51949)
I20240502 01:58:41.608389 24651 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 69182e5f4ae7401f89d22f063fa8df63 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:51949: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-69182e5f4ae7401f89d22f063fa8df63
I20240502 01:58:41.609225 24651 tablet_copy_source_session.cc:205] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.610056 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69182e5f4ae7401f89d22f063fa8df63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.612072 28137 tablet_copy_client.cc:783] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.612222 28137 tablet_copy_client.cc:647] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.613801 28158 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.615687 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-e720d8363bb049938798ab0b9aabc28d received from {username='slave'} at 127.17.243.5:52035
I20240502 01:58:41.615764 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-e720d8363bb049938798ab0b9aabc28d on tablet e720d8363bb049938798ab0b9aabc28d with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:41.623813 28288 tablet_copy_client.cc:523] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.625229 28288 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.629858 27576 raft_consensus.cc:1203] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.641084 28165 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.641410 28165 tablet_bootstrap.cc:492] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:41.641639 28165 ts_tablet_manager.cc:1367] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.088s user 0.024s sys 0.000s
I20240502 01:58:41.644738 28165 raft_consensus.cc:348] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.644901 28165 raft_consensus.cc:729] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:41.645025 28165 consensus_queue.cc:260] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:41.646749 24485 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:41.651789 27555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:41.651885 28350 ts_tablet_manager.cc:902] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.652462 28350 tablet_copy_client.cc:308] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.652578 28165 ts_tablet_manager.cc:1398] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:58:41.654397 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 752d360cfdd64846b6cd37c9b859d7a8 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:51949)
I20240502 01:58:41.654512 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 752d360cfdd64846b6cd37c9b859d7a8 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:51949: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-752d360cfdd64846b6cd37c9b859d7a8
I20240502 01:58:41.655532 24652 tablet_copy_source_session.cc:205] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.655850 25011 tablet_service.cc:1467] Processing CreateTablet for tablet af2eb5798c3843d29b9b6e353d850b24 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.655848 27289 tablet_service.cc:1467] Processing CreateTablet for tablet c6186e8ffddc47a4a3b428fcd7ce19fb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.656121 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6186e8ffddc47a4a3b428fcd7ce19fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.656076 27155 tablet_service.cc:1467] Processing CreateTablet for tablet af2eb5798c3843d29b9b6e353d850b24 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.656149 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2eb5798c3843d29b9b6e353d850b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.656322 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2eb5798c3843d29b9b6e353d850b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.657868 27644 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.658736 27644 tablet_bootstrap.cc:654] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.659394 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-26422e12ddf64e27b3913c60800e5a56 received from {username='slave'} at 127.17.243.5:47383
I20240502 01:58:41.659475 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-26422e12ddf64e27b3913c60800e5a56 on tablet 26422e12ddf64e27b3913c60800e5a56 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:41.659526 28357 tablet_bootstrap.cc:492] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:41.659703 27289 tablet_service.cc:1467] Processing CreateTablet for tablet b6c09e8f421d4ae2adf7c9b63121943c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.659946 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c09e8f421d4ae2adf7c9b63121943c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.659950 24743 tablet_service.cc:1467] Processing CreateTablet for tablet c6186e8ffddc47a4a3b428fcd7ce19fb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.660109 28358 tablet_bootstrap.cc:492] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.660183 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6186e8ffddc47a4a3b428fcd7ce19fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.661547 28357 tablet_bootstrap.cc:654] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.662199 27424 tablet_service.cc:1467] Processing CreateTablet for tablet c6186e8ffddc47a4a3b428fcd7ce19fb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.662463 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6186e8ffddc47a4a3b428fcd7ce19fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.662575 28358 tablet_bootstrap.cc:654] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.664670 27425 tablet_service.cc:1467] Processing CreateTablet for tablet b6c09e8f421d4ae2adf7c9b63121943c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.664844 28357 tablet_bootstrap.cc:492] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:41.664878 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c09e8f421d4ae2adf7c9b63121943c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.664928 28357 ts_tablet_manager.cc:1367] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:41.665341 28357 raft_consensus.cc:348] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.665463 28357 raft_consensus.cc:374] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.665495 28357 raft_consensus.cc:729] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.665607 28357 consensus_queue.cc:260] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.666381 28137 tablet_copy_client.cc:523] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.666680 28357 ts_tablet_manager.cc:1398] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:41.667351 28197 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.667416 28358 tablet_bootstrap.cc:492] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:41.667500 28358 ts_tablet_manager.cc:1367] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:58:41.667507 28137 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.667966 28358 raft_consensus.cc:348] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.668154 28358 raft_consensus.cc:374] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.668291 28197 tablet_bootstrap.cc:654] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.668450 28358 raft_consensus.cc:729] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.668519 27155 tablet_service.cc:1467] Processing CreateTablet for tablet b6c09e8f421d4ae2adf7c9b63121943c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.668663 28358 consensus_queue.cc:260] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.668743 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c09e8f421d4ae2adf7c9b63121943c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.669373 27555 tablet_service.cc:1467] Processing CreateTablet for tablet af2eb5798c3843d29b9b6e353d850b24 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.669607 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2eb5798c3843d29b9b6e353d850b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.671293 28358 ts_tablet_manager.cc:1398] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:41.673400 28358 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:41.674453 28358 tablet_bootstrap.cc:654] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.675940 28350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d360cfdd64846b6cd37c9b859d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.678086 28158 raft_consensus.cc:482] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.678171 28158 raft_consensus.cc:504] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.678640 28365 tablet_bootstrap.cc:492] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:41.679173 28274 ts_tablet_manager.cc:902] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:41.679721 28274 tablet_copy_client.cc:308] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:41.680178 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 9ede6fc65d044804848e6f61e8c238c0 from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:59275)
I20240502 01:58:41.680280 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 9ede6fc65d044804848e6f61e8c238c0 from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:59275: session id = 1049b4c8783b48f39be22bd113ad349f-9ede6fc65d044804848e6f61e8c238c0
I20240502 01:58:41.680277 28158 leader_election.cc:288] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:41.680814 28350 tablet_copy_client.cc:783] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.680958 28350 tablet_copy_client.cc:647] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.681236 24652 tablet_copy_source_session.cc:205] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:41.681728 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2eb5798c3843d29b9b6e353d850b24" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:41.681865 25032 raft_consensus.cc:2454] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:41.682094 28274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ede6fc65d044804848e6f61e8c238c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.682343 27110 leader_election.cc:302] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:41.682595 28158 raft_consensus.cc:2781] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.682658 28158 raft_consensus.cc:482] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.682698 28158 raft_consensus.cc:3037] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.683737 28158 raft_consensus.cc:504] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.684140 28158 leader_election.cc:288] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:41.684159 28274 tablet_copy_client.cc:783] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:41.684283 28274 tablet_copy_client.cc:647] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:41.684800 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2eb5798c3843d29b9b6e353d850b24" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:41.685575 27644 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:41.685659 27644 ts_tablet_manager.cc:1367] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
W20240502 01:58:41.685626 27113 leader_election.cc:341] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:41.686091 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2eb5798c3843d29b9b6e353d850b24" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:41.686115 27644 raft_consensus.cc:348] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.686198 25032 raft_consensus.cc:3037] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.686215 27644 raft_consensus.cc:374] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.686246 27644 raft_consensus.cc:729] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.686362 27644 consensus_queue.cc:260] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.686632 27644 ts_tablet_manager.cc:1398] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.686787 27644 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.687101 25032 raft_consensus.cc:2454] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:41.687394 27110 leader_election.cc:302] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters: bb653087e8554c7591cf547dd969f3be
I20240502 01:58:41.687644 27644 tablet_bootstrap.cc:654] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.687687 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2eb5798c3843d29b9b6e353d850b24" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
W20240502 01:58:41.688027 27113 leader_election.cc:341] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:41.688230 28158 raft_consensus.cc:2781] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.688311 28158 raft_consensus.cc:686] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:41.688449 28158 consensus_queue.cc:237] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.690505 28197 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:41.690621 28197 ts_tablet_manager.cc:1367] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240502 01:58:41.690670 24481 catalog_manager.cc:5484] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:41.691267 28197 raft_consensus.cc:348] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.691377 28197 raft_consensus.cc:374] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.691429 28197 raft_consensus.cc:729] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.691551 28197 consensus_queue.cc:260] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.691828 28197 ts_tablet_manager.cc:1398] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:41.691999 28197 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.692612 28365 tablet_bootstrap.cc:654] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.692863 28197 tablet_bootstrap.cc:654] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.694056 28358 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:41.694232 28358 ts_tablet_manager.cc:1367] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:58:41.696108 28358 raft_consensus.cc:348] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.696251 28358 raft_consensus.cc:374] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.696321 28358 raft_consensus.cc:729] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.696501 28358 consensus_queue.cc:260] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.698029 28197 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:41.698107 28197 ts_tablet_manager.cc:1367] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:41.698362 28358 ts_tablet_manager.cc:1398] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:41.698575 28197 raft_consensus.cc:348] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.698673 28197 raft_consensus.cc:374] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.698704 28197 raft_consensus.cc:729] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.698822 28197 consensus_queue.cc:260] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.699117 28197 ts_tablet_manager.cc:1398] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.700381 28366 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:41.701305 28366 tablet_bootstrap.cc:654] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.702693 28350 tablet_copy_client.cc:523] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.703526 28274 tablet_copy_client.cc:523] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:41.703949 28350 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:41.704731 28274 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:41.708667 28366 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:41.708796 28366 ts_tablet_manager.cc:1367] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:41.709221 28366 raft_consensus.cc:348] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.709381 28366 raft_consensus.cc:374] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.709448 28366 raft_consensus.cc:729] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.709625 28366 consensus_queue.cc:260] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.709987 28366 ts_tablet_manager.cc:1398] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.713456 28137 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.713686 28137 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:41.713824 28137 ts_tablet_manager.cc:1367] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.000s
I20240502 01:58:41.714190 28137 raft_consensus.cc:348] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.714288 28137 raft_consensus.cc:729] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:41.714372 28137 consensus_queue.cc:260] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.714617 28137 ts_tablet_manager.cc:1398] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:41.715044 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-69182e5f4ae7401f89d22f063fa8df63 received from {username='slave'} at 127.17.243.7:51949
I20240502 01:58:41.715134 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-69182e5f4ae7401f89d22f063fa8df63 on tablet 69182e5f4ae7401f89d22f063fa8df63 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:41.720353 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:41.721140 28105 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.724732 27312 raft_consensus.cc:3037] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.725670 27312 raft_consensus.cc:1261] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.726210 28315 consensus_queue.cc:1035] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.728837 28288 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.728878 24481 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]). Expected version 1 got 0
I20240502 01:58:41.729071 28380 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:41.729377 28380 tablet.cc:1620] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:41.730260 27644 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:41.730327 28365 tablet_bootstrap.cc:492] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:41.730338 27644 ts_tablet_manager.cc:1367] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240502 01:58:41.730410 28365 ts_tablet_manager.cc:1367] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240502 01:58:41.730748 27644 raft_consensus.cc:348] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.730837 27644 raft_consensus.cc:374] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.730866 27644 raft_consensus.cc:729] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.730867 28365 raft_consensus.cc:348] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.730974 28365 raft_consensus.cc:374] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.731003 28365 raft_consensus.cc:729] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.730990 27644 consensus_queue.cc:260] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.731115 28365 consensus_queue.cc:260] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.731294 27644 ts_tablet_manager.cc:1398] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.731405 28365 ts_tablet_manager.cc:1398] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.731694 24481 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]). Expected version 1 got 0
I20240502 01:58:41.731825 24896 raft_consensus.cc:1261] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.732224 28105 tablet_bootstrap.cc:492] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:41.732187 27930 consensus_queue.cc:1035] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.732391 28105 ts_tablet_manager.cc:1367] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.128s user 0.025s sys 0.000s
I20240502 01:58:41.732777 28105 raft_consensus.cc:348] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.732872 28105 raft_consensus.cc:729] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:41.732964 28105 consensus_queue.cc:260] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:41.733191 28105 ts_tablet_manager.cc:1398] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.733655 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-aafc084daeec44d5aa2ac9fca5615484 received from {username='slave'} at 127.17.243.8:44271
I20240502 01:58:41.733728 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session bb653087e8554c7591cf547dd969f3be-aafc084daeec44d5aa2ac9fca5615484 on tablet aafc084daeec44d5aa2ac9fca5615484 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:41.733481 28288 tablet_bootstrap.cc:492] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:41.734222 28288 ts_tablet_manager.cc:1367] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.109s user 0.024s sys 0.003s
I20240502 01:58:41.734666 28288 raft_consensus.cc:348] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.734760 28288 raft_consensus.cc:729] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:41.734839 28288 consensus_queue.cc:260] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.735090 28288 ts_tablet_manager.cc:1398] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:41.735911 28381 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:41.736150 28381 tablet.cc:1620] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:41.736353 24486 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]). Expected version 1 got 0
I20240502 01:58:41.737010 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-a094659d539e4c4f9861e356168c5894 received from {username='slave'} at 127.17.243.6:59275
I20240502 01:58:41.737074 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-a094659d539e4c4f9861e356168c5894 on tablet a094659d539e4c4f9861e356168c5894 with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:41.742295 27941 raft_consensus.cc:482] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.742393 27941 raft_consensus.cc:504] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.742872 27941 leader_election.cc:288] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:41.742918 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d] alter complete (version 1)
I20240502 01:58:41.757462 28350 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.757831 28350 tablet_bootstrap.cc:492] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:41.757982 27930 raft_consensus.cc:482] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.758057 27930 raft_consensus.cc:504] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.758106 28350 ts_tablet_manager.cc:1367] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.000s
I20240502 01:58:41.758942 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6186e8ffddc47a4a3b428fcd7ce19fb" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:41.759064 27312 raft_consensus.cc:2454] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:41.764276 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6186e8ffddc47a4a3b428fcd7ce19fb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:41.764401 27312 raft_consensus.cc:2454] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:41.764778 24698 leader_election.cc:302] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:41.765076 27941 raft_consensus.cc:2781] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.765132 27941 raft_consensus.cc:482] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.765169 27941 raft_consensus.cc:3037] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.765831 28274 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:41.766088 28274 tablet_bootstrap.cc:492] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:41.766220 28274 ts_tablet_manager.cc:1367] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.062s user 0.021s sys 0.003s
I20240502 01:58:41.766183 27941 raft_consensus.cc:504] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.766549 27941 leader_election.cc:288] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:41.766600 28274 raft_consensus.cc:348] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.766695 28274 raft_consensus.cc:729] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:41.766777 28274 consensus_queue.cc:260] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.767279 28274 ts_tablet_manager.cc:1398] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.767381 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6186e8ffddc47a4a3b428fcd7ce19fb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:41.767482 27312 raft_consensus.cc:3037] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.768338 27312 raft_consensus.cc:2454] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:41.768935 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-9ede6fc65d044804848e6f61e8c238c0 received from {username='slave'} at 127.17.243.6:59275
I20240502 01:58:41.769014 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-9ede6fc65d044804848e6f61e8c238c0 on tablet 9ede6fc65d044804848e6f61e8c238c0 with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:41.769639 24698 leader_election.cc:302] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:41.769861 28387 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:41.770277 28387 tablet.cc:1620] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:41.770711 27941 raft_consensus.cc:2781] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.770733 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6186e8ffddc47a4a3b428fcd7ce19fb" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:41.770911 27941 raft_consensus.cc:686] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:41.771063 27941 consensus_queue.cc:237] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:41.758553 27930 leader_election.cc:288] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:41.760195 27377 leader_election.cc:302] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:41.761497 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6186e8ffddc47a4a3b428fcd7ce19fb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:41.773298 27446 raft_consensus.cc:2454] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:41.766957 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6186e8ffddc47a4a3b428fcd7ce19fb" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:41.773481 27445 raft_consensus.cc:3037] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.773703 24481 catalog_manager.cc:5484] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:41.774479 27445 raft_consensus.cc:2454] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:41.775056 27930 raft_consensus.cc:2741] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 01:58:41.776124 28350 raft_consensus.cc:348] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.776177 24896 consensus_queue.cc:237] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:41.776248 28350 raft_consensus.cc:729] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:41.776342 28350 consensus_queue.cc:260] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:41.777400 28350 ts_tablet_manager.cc:1398] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:41.777674 24765 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.778079 25032 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:41.778098 27093 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.778484 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-752d360cfdd64846b6cd37c9b859d7a8 received from {username='slave'} at 127.17.243.7:51949
I20240502 01:58:41.778481 27093 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:41.778576 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-752d360cfdd64846b6cd37c9b859d7a8 on tablet 752d360cfdd64846b6cd37c9b859d7a8 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:41.783090 28396 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.785549 24765 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.787343 24486 catalog_manager.cc:5484] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:41.789809 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 938581084b27405cbc01917e1a982208. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:41.791268 25032 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:41.795240 27578 raft_consensus.cc:1203] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.812501 27177 raft_consensus.cc:1203] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.818977 27177 raft_consensus.cc:1203] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.842388 27578 raft_consensus.cc:1203] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.845621 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:41.851222 28380 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:41.851526 28380 tablet.cc:1620] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:41.851645 28387 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:41.851930 28387 tablet.cc:1620] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:41.851925 24481 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d]). Expected version 2 got 1
I20240502 01:58:41.853123 28381 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:41.853394 28381 tablet.cc:1620] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:41.857573 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=2abb7ddddba145ecae312c47aaf5208d] alter complete (version 2)
I20240502 01:58:41.867225 27312 raft_consensus.cc:1203] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.909358 27445 raft_consensus.cc:1203] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.922348 27177 raft_consensus.cc:1203] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.944743 27312 raft_consensus.cc:1203] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.951638 28315 raft_consensus.cc:482] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.951756 28315 raft_consensus.cc:504] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.952252 28315 leader_election.cc:288] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:41.952548 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c09e8f421d4ae2adf7c9b63121943c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:41.952678 27312 raft_consensus.cc:2454] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:41.952968 27377 leader_election.cc:302] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:41.953136 28315 raft_consensus.cc:2781] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.953208 28315 raft_consensus.cc:482] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.953246 28315 raft_consensus.cc:3037] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.954259 28315 raft_consensus.cc:504] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.954605 28315 leader_election.cc:288] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:41.954990 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c09e8f421d4ae2adf7c9b63121943c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:41.955092 27312 raft_consensus.cc:3037] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.956017 27312 raft_consensus.cc:2454] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:41.956260 27377 leader_election.cc:302] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:41.957019 28315 raft_consensus.cc:2781] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.957105 28315 raft_consensus.cc:686] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:41.957229 28315 consensus_queue.cc:237] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:41.957813 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c09e8f421d4ae2adf7c9b63121943c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:41.957842 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c09e8f421d4ae2adf7c9b63121943c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:41.957973 27177 raft_consensus.cc:2454] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:41.958842 24481 catalog_manager.cc:5484] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:41.961122 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:41.964520 27578 raft_consensus.cc:1203] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.966889 27867 raft_consensus.cc:482] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.966981 27867 raft_consensus.cc:504] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:41.967396 27867 leader_election.cc:288] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:41.968094 28380 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:41.969096 28381 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:41.969173 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2eb5798c3843d29b9b6e353d850b24" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:41.969348 25034 raft_consensus.cc:2379] T af2eb5798c3843d29b9b6e353d850b24 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb653087e8554c7591cf547dd969f3be in current term 1: Already voted for candidate 9dd619a595e94004ac0f7f5a9054e7cc in this term.
I20240502 01:58:41.969992 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2eb5798c3843d29b9b6e353d850b24" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:41.971627 27513 leader_election.cc:302] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb653087e8554c7591cf547dd969f3be; no voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:58:41.973728 27867 raft_consensus.cc:3037] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.974257 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=2abb7ddddba145ecae312c47aaf5208d] alter complete (version 3)
I20240502 01:58:41.974236 28387 tablet.cc:1722] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:41.975069 27867 raft_consensus.cc:2726] T af2eb5798c3843d29b9b6e353d850b24 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:58:41.978144 27176 raft_consensus.cc:1203] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:41.988519 27445 raft_consensus.cc:1203] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.013366 24487 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 01:58:42.013504 24487 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 01:58:42.015293 27894 raft_consensus.cc:1050] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:42.015827 27894 consensus_queue.cc:237] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.017218 24487 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet a6846b4da6654802b00203529a01e398
I20240502 01:58:42.017446 24632 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.017805 24487 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet abfd63f6a0e84f3e855e7c07d3b9cc70
I20240502 01:58:42.017871 27890 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.017990 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet a6846b4da6654802b00203529a01e398 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.018222 28027 tablet_replica.cc:330] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.018325 24487 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet aafc084daeec44d5aa2ac9fca5615484
I20240502 01:58:42.018512 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.018528 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet a6846b4da6654802b00203529a01e398 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:42.018695 27740 tablet_replica.cc:330] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.018817 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet aafc084daeec44d5aa2ac9fca5615484 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.018819 27740 raft_consensus.cc:2229] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.019019 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet a6846b4da6654802b00203529a01e398 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.019102 27740 raft_consensus.cc:2258] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.019105 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.019186 27864 tablet_replica.cc:330] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.019245 28027 raft_consensus.cc:2229] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.019304 27864 raft_consensus.cc:2229] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.019376 27578 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.019395 28027 raft_consensus.cc:2258] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.019444 27864 raft_consensus.cc:2258] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.019637 27740 ts_tablet_manager.cc:1875] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.019883 28027 ts_tablet_manager.cc:1875] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.019938 27864 ts_tablet_manager.cc:1875] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.020045 27890 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.021404 28535 tablet_replica.cc:330] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:42.021526 28535 raft_consensus.cc:2229] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:42.021638 28535 raft_consensus.cc:2258] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:42.021811 27740 ts_tablet_manager.cc:1888] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.021873 27740 log.cc:1196] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/abfd63f6a0e84f3e855e7c07d3b9cc70
I20240502 01:58:42.022006 27864 ts_tablet_manager.cc:1888] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.022065 27864 log.cc:1196] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6846b4da6654802b00203529a01e398
I20240502 01:58:42.022150 24896 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.022195 28535 ts_tablet_manager.cc:1875] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.022320 27864 ts_tablet_manager.cc:1909] T a6846b4da6654802b00203529a01e398 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:42.022532 27890 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.023003 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet aafc084daeec44d5aa2ac9fca5615484 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.023053 27555 tablet_service.cc:1514] Processing DeleteTablet for tablet aafc084daeec44d5aa2ac9fca5615484 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:42.023190 27740 ts_tablet_manager.cc:1909] T abfd63f6a0e84f3e855e7c07d3b9cc70 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:42.023770 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet a6846b4da6654802b00203529a01e398 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.024058 27740 tablet_replica.cc:330] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.024158 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.024338 27864 tablet_replica.cc:330] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.024370 27740 raft_consensus.cc:2229] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.024513 27740 raft_consensus.cc:2258] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.024750 27864 raft_consensus.cc:2229] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.024814 28027 ts_tablet_manager.cc:1888] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.024884 28027 log.cc:1196] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6846b4da6654802b00203529a01e398
I20240502 01:58:42.024930 27740 ts_tablet_manager.cc:1875] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.024933 27864 raft_consensus.cc:2258] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.025130 28027 ts_tablet_manager.cc:1909] T a6846b4da6654802b00203529a01e398 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:42.025465 27864 ts_tablet_manager.cc:1875] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.025811 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet a6846b4da6654802b00203529a01e398 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.026057 28027 tablet_replica.cc:330] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.026202 28535 ts_tablet_manager.cc:1888] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.026232 28027 raft_consensus.cc:2229] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.026329 28535 log.cc:1196] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/a6846b4da6654802b00203529a01e398
I20240502 01:58:42.026392 28027 raft_consensus.cc:2258] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.026640 28535 ts_tablet_manager.cc:1909] T a6846b4da6654802b00203529a01e398 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:42.026948 28027 ts_tablet_manager.cc:1875] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.027185 27864 ts_tablet_manager.cc:1888] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.027185 27740 ts_tablet_manager.cc:1888] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.027252 27740 log.cc:1196] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/aafc084daeec44d5aa2ac9fca5615484
I20240502 01:58:42.027390 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet a6846b4da6654802b00203529a01e398 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.027521 28538 tablet_replica.cc:330] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:42.027550 27740 ts_tablet_manager.cc:1909] T aafc084daeec44d5aa2ac9fca5615484 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:42.027565 27864 log.cc:1196] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/abfd63f6a0e84f3e855e7c07d3b9cc70
I20240502 01:58:42.027871 28538 raft_consensus.cc:2229] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:42.028057 28538 raft_consensus.cc:2258] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:42.028091 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet aafc084daeec44d5aa2ac9fca5615484 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.028600 27864 ts_tablet_manager.cc:1909] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:42.028970 28027 ts_tablet_manager.cc:1888] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.029040 28027 log.cc:1196] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/aafc084daeec44d5aa2ac9fca5615484
I20240502 01:58:42.029088 28538 ts_tablet_manager.cc:1875] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.029282 28027 ts_tablet_manager.cc:1909] T aafc084daeec44d5aa2ac9fca5615484 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:42.029330 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.029871 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet aafc084daeec44d5aa2ac9fca5615484 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.030786 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet a6846b4da6654802b00203529a01e398 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.031400 28540 tablet_replica.cc:330] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.031446 28538 ts_tablet_manager.cc:1888] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.031594 28538 log.cc:1196] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/aafc084daeec44d5aa2ac9fca5615484
I20240502 01:58:42.031853 28540 raft_consensus.cc:2229] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.032151 28540 raft_consensus.cc:2258] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.032655 28540 ts_tablet_manager.cc:1875] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.032660 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:42.032845 28300 tablet_replica.cc:330] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:42.032980 28300 raft_consensus.cc:2229] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:42.033092 28300 raft_consensus.cc:2258] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:42.033607 28300 ts_tablet_manager.cc:1875] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.034335 28538 ts_tablet_manager.cc:1909] T aafc084daeec44d5aa2ac9fca5615484 P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:42.035666 28300 ts_tablet_manager.cc:1888] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.035698 28540 ts_tablet_manager.cc:1888] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.035730 28300 log.cc:1196] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/abfd63f6a0e84f3e855e7c07d3b9cc70
I20240502 01:58:42.035810 28540 log.cc:1196] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/a6846b4da6654802b00203529a01e398
I20240502 01:58:42.035974 28300 ts_tablet_manager.cc:1909] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:42.036100 28540 ts_tablet_manager.cc:1909] T a6846b4da6654802b00203529a01e398 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:42.035904 27578 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.036772 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet aafc084daeec44d5aa2ac9fca5615484 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.035501 28534 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.037719 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet abfd63f6a0e84f3e855e7c07d3b9cc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.037741 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet aafc084daeec44d5aa2ac9fca5615484 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.038690 24486 catalog_manager.cc:5484] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.038690 24897 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.039116 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.039227 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet a6846b4da6654802b00203529a01e398 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.039829 28540 tablet_replica.cc:330] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.039794 24630 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.039985 28540 raft_consensus.cc:2229] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.040099 28540 raft_consensus.cc:2258] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.040575 28540 ts_tablet_manager.cc:1875] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.043737 25034 consensus_queue.cc:237] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.045897 24630 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.046154 27578 raft_consensus.cc:1261] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.046299 27890 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.046489 28534 consensus_queue.cc:1035] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.046692 28540 ts_tablet_manager.cc:1888] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.046767 28540 log.cc:1196] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/abfd63f6a0e84f3e855e7c07d3b9cc70
I20240502 01:58:42.048131 27017 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.049501 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce35d9953e294b4986ff9e93e12db6f6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.048607 24630 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.050057 28540 ts_tablet_manager.cc:1909] T abfd63f6a0e84f3e855e7c07d3b9cc70 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:42.050034 27312 raft_consensus.cc:1203] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.050875 28540 tablet_replica.cc:330] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.051199 28540 raft_consensus.cc:2229] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.051468 28540 raft_consensus.cc:2258] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.051692 27578 raft_consensus.cc:2932] T ce35d9953e294b4986ff9e93e12db6f6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.052532 24487 catalog_manager.cc:5484] T ce35d9953e294b4986ff9e93e12db6f6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.055760 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet ce35d9953e294b4986ff9e93e12db6f6 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.056054 28027 tablet_replica.cc:330] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.056167 28027 raft_consensus.cc:2229] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.056311 28027 raft_consensus.cc:2258] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.056967 28540 ts_tablet_manager.cc:1875] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.058137 28027 ts_tablet_manager.cc:1875] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.060182 28027 ts_tablet_manager.cc:1888] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.060245 28027 log.cc:1196] T ce35d9953e294b4986ff9e93e12db6f6 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/ce35d9953e294b4986ff9e93e12db6f6
I20240502 01:58:42.060730 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet ce35d9953e294b4986ff9e93e12db6f6 (table rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]) successfully deleted
I20240502 01:58:42.061483 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet abfd63f6a0e84f3e855e7c07d3b9cc70 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.063560 28540 ts_tablet_manager.cc:1888] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.065754 28540 log.cc:1196] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/aafc084daeec44d5aa2ac9fca5615484
I20240502 01:58:42.066452 28540 ts_tablet_manager.cc:1909] T aafc084daeec44d5aa2ac9fca5615484 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:42.067440 27289 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:42.067986 27425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:42.070207 27155 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:42.072044 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet aafc084daeec44d5aa2ac9fca5615484 (table rebalance_test_table_8 [id=32cc0427d92e4b16b64c4ffe48e87ae1]) successfully deleted
I20240502 01:58:42.079418 24481 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:42.079540 24481 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:42.081777 24481 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7
I20240502 01:58:42.082442 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:42.082496 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.082620 28535 tablet_replica.cc:330] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:42.082657 28027 tablet_replica.cc:330] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.082677 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:42.082736 28535 raft_consensus.cc:2229] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.082788 28027 raft_consensus.cc:2229] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.082839 28535 raft_consensus.cc:2258] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.082897 28027 raft_consensus.cc:2258] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.083146 28553 tablet_replica.cc:330] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:42.083303 28553 raft_consensus.cc:2229] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.083477 28535 ts_tablet_manager.cc:1875] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.083602 28027 ts_tablet_manager.cc:1875] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.083622 28553 raft_consensus.cc:2258] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.084342 28553 ts_tablet_manager.cc:1875] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.085559 28027 ts_tablet_manager.cc:1888] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.085561 28535 ts_tablet_manager.cc:1888] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.085619 28027 log.cc:1196] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/5fa2bc99e2ba4fc5998f6af3e5ac47c7
I20240502 01:58:42.085623 28535 log.cc:1196] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/5fa2bc99e2ba4fc5998f6af3e5ac47c7
I20240502 01:58:42.085860 28535 ts_tablet_manager.cc:1909] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:42.085865 28027 ts_tablet_manager.cc:1909] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:42.086858 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table_new_name [id=2abb7ddddba145ecae312c47aaf5208d]) successfully deleted
I20240502 01:58:42.087097 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table_new_name [id=2abb7ddddba145ecae312c47aaf5208d]) successfully deleted
I20240502 01:58:42.088111 28553 ts_tablet_manager.cc:1888] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.088229 28553 log.cc:1196] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/5fa2bc99e2ba4fc5998f6af3e5ac47c7
I20240502 01:58:42.088650 28553 ts_tablet_manager.cc:1909] T 5fa2bc99e2ba4fc5998f6af3e5ac47c7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:42.088981 24877 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:58:42.090333 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 5fa2bc99e2ba4fc5998f6af3e5ac47c7 (table rebalancer_dynamic_table_new_name [id=2abb7ddddba145ecae312c47aaf5208d]) successfully deleted
I20240502 01:58:42.091519 24744 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:58:42.093869 25011 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:58:42.095556 24611 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:58:42.096411 27555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:42.113896 27578 raft_consensus.cc:1203] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.170967 24897 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:42.172147 24765 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.172148 25034 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.172502 28396 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.172691 27093 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:42.172889 24833 consensus_peers.cc:485] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ec1158a7414039b6f4bc855207c938. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.174350 27093 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:42.174907 24765 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:42.175031 25034 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:42.176955 24481 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.183398 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:42.187261 27445 raft_consensus.cc:1203] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.188016 28404 raft_consensus.cc:1050] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:42.188504 28404 consensus_queue.cc:237] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.189677 27155 tablet_service.cc:1467] Processing CreateTablet for tablet c95fa6f5402d42d09a68f3793b1f000f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]), partition=
I20240502 01:58:42.189918 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95fa6f5402d42d09a68f3793b1f000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.189910 27289 tablet_service.cc:1467] Processing CreateTablet for tablet c95fa6f5402d42d09a68f3793b1f000f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]), partition=
I20240502 01:58:42.190086 24630 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.190133 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95fa6f5402d42d09a68f3793b1f000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.190279 24897 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.190583 28393 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.190752 27941 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.191001 27578 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.191114 27555 tablet_service.cc:1467] Processing CreateTablet for tablet c95fa6f5402d42d09a68f3793b1f000f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]), partition=
I20240502 01:58:42.191352 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95fa6f5402d42d09a68f3793b1f000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.191569 27941 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.192605 24765 consensus_queue.cc:237] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:42.194072 25034 raft_consensus.cc:1261] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.194097 28365 tablet_bootstrap.cc:492] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:42.194450 27644 tablet_bootstrap.cc:492] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:42.194736 27941 consensus_queue.cc:1035] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.194875 28365 tablet_bootstrap.cc:654] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.194943 27445 raft_consensus.cc:1203] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.195295 27644 tablet_bootstrap.cc:654] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:42.195791 24700 consensus_peers.cc:485] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f999b09b79a4632936833f351c4b28b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.196969 24632 raft_consensus.cc:1261] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.197333 28568 consensus_queue.cc:1035] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.197842 28358 tablet_bootstrap.cc:492] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:42.198613 27644 tablet_bootstrap.cc:492] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:42.198705 27644 ts_tablet_manager.cc:1367] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:42.198772 28358 tablet_bootstrap.cc:654] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.199107 27644 raft_consensus.cc:348] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.199203 27644 raft_consensus.cc:374] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.199234 27644 raft_consensus.cc:729] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.199357 27644 consensus_queue.cc:260] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.199611 27644 ts_tablet_manager.cc:1398] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.200592 28358 tablet_bootstrap.cc:492] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:42.200666 28358 ts_tablet_manager.cc:1367] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:42.200990 28358 raft_consensus.cc:348] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.201081 28358 raft_consensus.cc:374] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.200949 28568 raft_consensus.cc:2932] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:42.201107 28358 raft_consensus.cc:729] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.201205 28358 consensus_queue.cc:260] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.201431 28358 ts_tablet_manager.cc:1398] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.202651 28365 tablet_bootstrap.cc:492] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:42.202723 28365 ts_tablet_manager.cc:1367] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:42.203049 28365 raft_consensus.cc:348] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.203145 28365 raft_consensus.cc:374] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.203171 28365 raft_consensus.cc:729] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.203143 24632 raft_consensus.cc:2932] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:42.203270 28365 consensus_queue.cc:260] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.203482 28365 ts_tablet_manager.cc:1398] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.205229 24484 catalog_manager.cc:5484] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.206154 28393 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.206593 24630 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.208762 24897 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.210376 25034 raft_consensus.cc:2932] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:42.213007 27578 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.213587 24484 catalog_manager.cc:5484] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.219185 24765 consensus_queue.cc:237] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.220600 24630 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.220939 27941 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.220988 27578 raft_consensus.cc:1261] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.221581 27941 consensus_queue.cc:1035] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.222877 27941 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.223156 24630 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.223618 27578 raft_consensus.cc:2932] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.224514 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70d0c3a7075a4e40a621b73931a3a33b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.226065 24487 catalog_manager.cc:5484] T 70d0c3a7075a4e40a621b73931a3a33b P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.228689 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d0c3a7075a4e40a621b73931a3a33b with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.228929 28027 tablet_replica.cc:330] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.229080 28027 raft_consensus.cc:2229] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.229262 28027 raft_consensus.cc:2258] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.230026 28027 ts_tablet_manager.cc:1875] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.231931 28027 ts_tablet_manager.cc:1888] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.231990 28027 log.cc:1196] T 70d0c3a7075a4e40a621b73931a3a33b P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/70d0c3a7075a4e40a621b73931a3a33b
I20240502 01:58:42.232406 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 70d0c3a7075a4e40a621b73931a3a33b (table rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]) successfully deleted
I20240502 01:58:42.233098 28274 ts_tablet_manager.cc:902] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:42.233551 28274 tablet_copy_client.cc:308] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:42.234303 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 938581084b27405cbc01917e1a982208 from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:44697)
I20240502 01:58:42.234385 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 938581084b27405cbc01917e1a982208 from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:44697: session id = 1049b4c8783b48f39be22bd113ad349f-938581084b27405cbc01917e1a982208
I20240502 01:58:42.235208 24918 tablet_copy_source_session.cc:205] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:42.236269 27926 consensus_queue.cc:1035] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.236584 28274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938581084b27405cbc01917e1a982208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.238765 28274 tablet_copy_client.cc:783] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:42.238905 28274 tablet_copy_client.cc:647] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:42.240371 27926 consensus_queue.cc:1035] T af2eb5798c3843d29b9b6e353d850b24 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.242049 28274 tablet_copy_client.cc:523] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:42.243161 28274 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:42.263355 28274 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:42.263577 28274 tablet_bootstrap.cc:492] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:42.263872 28274 ts_tablet_manager.cc:1367] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240502 01:58:42.264313 28274 raft_consensus.cc:348] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:42.264659 28274 raft_consensus.cc:729] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:42.264798 28274 consensus_queue.cc:260] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:42.265226 28274 ts_tablet_manager.cc:1398] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:42.265734 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-938581084b27405cbc01917e1a982208 received from {username='slave'} at 127.17.243.6:44697
I20240502 01:58:42.265810 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-938581084b27405cbc01917e1a982208 on tablet 938581084b27405cbc01917e1a982208 with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:42.281586 28568 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.287480 28568 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.289897 27311 raft_consensus.cc:1203] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.294751 28534 raft_consensus.cc:1050] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:42.295138 28534 consensus_queue.cc:237] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:42.296554 27176 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.296762 24897 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.296972 27017 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.297537 27017 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.297763 24630 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.298439 27017 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.300734 27890 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.301156 24897 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.302657 24630 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.303094 27176 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.304019 24486 catalog_manager.cc:5484] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.311914 25034 consensus_queue.cc:237] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:42.312994 27176 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.313437 27894 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.314461 24897 raft_consensus.cc:1261] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.315555 27894 consensus_queue.cc:1035] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.317050 27894 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.317368 27176 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.318420 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0345ae8459544e049edf0ef0bcc5136d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.319682 24484 catalog_manager.cc:5484] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.321915 24897 raft_consensus.cc:2932] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.322177 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345ae8459544e049edf0ef0bcc5136d with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.323242 28540 tablet_replica.cc:330] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.323385 28540 raft_consensus.cc:2229] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.323530 28540 raft_consensus.cc:2258] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.325846 28540 ts_tablet_manager.cc:1875] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.328064 28540 ts_tablet_manager.cc:1888] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.328140 28540 log.cc:1196] T 0345ae8459544e049edf0ef0bcc5136d P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/0345ae8459544e049edf0ef0bcc5136d
I20240502 01:58:42.328998 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 0345ae8459544e049edf0ef0bcc5136d (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.350157 27017 raft_consensus.cc:1050] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:42.350476 27017 consensus_queue.cc:237] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.351722 24630 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.352069 28534 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.352296 24897 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.352421 27578 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.352659 28534 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.353061 28534 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.354975 27894 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.356456 24632 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.356454 24897 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.358502 24481 catalog_manager.cc:5484] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.359161 27578 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.364745 25034 consensus_queue.cc:237] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.365736 27578 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.365742 24897 raft_consensus.cc:1261] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.366078 27894 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.366222 28534 consensus_queue.cc:1035] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.367595 27890 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.368000 24897 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.368068 27578 raft_consensus.cc:2932] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.368896 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61a7e0e4e54f47849bb29d1e7ae029af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.369385 24765 consensus_queue.cc:237] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.370338 24485 catalog_manager.cc:5484] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.370868 24897 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.370987 24632 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.371323 28393 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.371471 27941 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.373227 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.373414 28540 tablet_replica.cc:330] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
W20240502 01:58:42.373517 24701 consensus_peers.cc:485] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ac6a4310330499483182c93827343c0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.373564 28540 raft_consensus.cc:2229] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.373638 27941 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.373747 28540 raft_consensus.cc:2258] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.374390 24632 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.374644 28540 ts_tablet_manager.cc:1875] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.376551 24487 catalog_manager.cc:5484] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.376570 28540 ts_tablet_manager.cc:1888] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.376787 28540 log.cc:1196] T 61a7e0e4e54f47849bb29d1e7ae029af P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/61a7e0e4e54f47849bb29d1e7ae029af
I20240502 01:58:42.377718 24897 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.378063 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 61a7e0e4e54f47849bb29d1e7ae029af (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:42.385442 25034 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.386786 24765 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:42.386778 24966 consensus_peers.cc:485] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 488816b0d5d542bcb5fa12ff7e70fd40. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.386794 24897 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.387120 27894 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.387378 27017 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.389225 27894 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.389647 24764 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.391916 24487 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.393107 24897 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.394555 27645 raft_consensus.cc:1050] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:42.395001 27645 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.396508 24632 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.396699 25034 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.397060 28396 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.398340 28396 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.400278 27445 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.400768 28396 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.401897 28396 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.403154 24632 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.403173 27445 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.403398 25034 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.403546 24485 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.443303 28315 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.446018 27176 raft_consensus.cc:3037] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.447350 28447 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.464641 27645 raft_consensus.cc:1050] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:42.464708 27017 raft_consensus.cc:1050] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:42.465011 27645 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.465087 27017 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:42.466609 24764 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.466701 27445 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.466699 27311 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.467077 27894 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.467500 25034 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.467661 24897 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.468111 28534 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.468395 28534 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.469286 27093 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.470149 27093 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.471765 24765 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.472333 27093 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.473428 27093 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.475437 24765 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.475898 27865 raft_consensus.cc:482] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:42.476003 27865 raft_consensus.cc:504] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.476405 27865 leader_election.cc:288] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:42.476667 24487 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.476927 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95fa6f5402d42d09a68f3793b1f000f" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:42.477070 27311 raft_consensus.cc:2454] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:42.477280 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95fa6f5402d42d09a68f3793b1f000f" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:42.477396 27176 raft_consensus.cc:2454] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:42.477339 27445 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.477592 25034 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.477787 27513 leader_election.cc:302] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:42.478901 27865 raft_consensus.cc:2781] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:42.478972 27865 raft_consensus.cc:482] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:42.479004 27865 raft_consensus.cc:3037] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.479159 27894 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.479821 27865 raft_consensus.cc:504] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.480139 27865 leader_election.cc:288] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:42.480883 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95fa6f5402d42d09a68f3793b1f000f" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:42.480989 27311 raft_consensus.cc:3037] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.481889 27311 raft_consensus.cc:2454] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:42.482124 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95fa6f5402d42d09a68f3793b1f000f" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:42.482241 27176 raft_consensus.cc:3037] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.482666 24487 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.483193 27176 raft_consensus.cc:2454] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:42.483639 27510 leader_election.cc:302] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:42.483829 27865 raft_consensus.cc:2781] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:42.483887 27865 raft_consensus.cc:686] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:42.483987 27865 consensus_queue.cc:237] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.484196 24896 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.485388 27445 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.481693 27312 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.486071 24481 catalog_manager.cc:5484] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.486198 24764 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.490083 27645 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.490250 24897 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.493891 27645 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.494683 25034 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:42.495355 27445 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.495745 27312 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.496028 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:42.496150 27017 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.496610 24896 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.496976 27017 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.498114 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91d0d9525b5f4384861dc50d0bf4967d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.498256 27890 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.498591 24484 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.499588 24765 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.500171 28779 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.500258 24896 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.501294 27312 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.501534 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26b7501639064035a570b3b4c1cf083f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.501478 24485 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.502617 28404 raft_consensus.cc:1050] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:42.502990 28404 consensus_queue.cc:237] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:42.504012 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d0d9525b5f4384861dc50d0bf4967d with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.504233 27740 tablet_replica.cc:330] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.504366 27740 raft_consensus.cc:2229] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.504487 25034 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.504541 27740 raft_consensus.cc:2258] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.505337 27740 ts_tablet_manager.cc:1875] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.505589 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b7501639064035a570b3b4c1cf083f with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.506057 27864 tablet_replica.cc:330] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.506175 27864 raft_consensus.cc:2229] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.506294 27864 raft_consensus.cc:2258] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.506415 28393 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.506916 27176 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.506948 24896 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.507378 27740 ts_tablet_manager.cc:1888] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.507439 27740 log.cc:1196] T 91d0d9525b5f4384861dc50d0bf4967d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/91d0d9525b5f4384861dc50d0bf4967d
I20240502 01:58:42.507450 24765 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.507467 27864 ts_tablet_manager.cc:1875] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.508739 28404 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.509451 27864 ts_tablet_manager.cc:1888] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.509512 27864 log.cc:1196] T 26b7501639064035a570b3b4c1cf083f P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/26b7501639064035a570b3b4c1cf083f
I20240502 01:58:42.510110 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 26b7501639064035a570b3b4c1cf083f (table rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]) successfully deleted
I20240502 01:58:42.511096 28393 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.512306 27425 tablet_service.cc:1467] Processing CreateTablet for tablet a85e6a0806654abd81ca5661a2d019ce (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.512553 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e6a0806654abd81ca5661a2d019ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.513249 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 584cbed9fc494fc2b32c1bfd2571d88b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.513484 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584cbed9fc494fc2b32c1bfd2571d88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.513489 27424 tablet_service.cc:1467] Processing CreateTablet for tablet adc992deb712458aa793eaa11019de1f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.513720 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc992deb712458aa793eaa11019de1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.513710 27557 tablet_service.cc:1467] Processing CreateTablet for tablet adc992deb712458aa793eaa11019de1f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.513916 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc992deb712458aa793eaa11019de1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.515412 28365 tablet_bootstrap.cc:492] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:42.515633 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 584cbed9fc494fc2b32c1bfd2571d88b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.513494 27555 tablet_service.cc:1467] Processing CreateTablet for tablet a85e6a0806654abd81ca5661a2d019ce (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.515856 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e6a0806654abd81ca5661a2d019ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.515892 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584cbed9fc494fc2b32c1bfd2571d88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.516134 28365 tablet_bootstrap.cc:654] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.516978 28365 tablet_bootstrap.cc:492] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:42.517057 28365 ts_tablet_manager.cc:1367] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:42.517402 28365 raft_consensus.cc:348] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.517494 28365 raft_consensus.cc:374] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.517524 28365 raft_consensus.cc:729] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.517642 28365 consensus_queue.cc:260] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.517894 28365 ts_tablet_manager.cc:1398] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.518105 28365 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:42.518209 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 584cbed9fc494fc2b32c1bfd2571d88b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.518440 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584cbed9fc494fc2b32c1bfd2571d88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.518899 28365 tablet_bootstrap.cc:654] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.519038 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 91d0d9525b5f4384861dc50d0bf4967d (table rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]) successfully deleted
I20240502 01:58:42.519388 27289 tablet_service.cc:1467] Processing CreateTablet for tablet a85e6a0806654abd81ca5661a2d019ce (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.519634 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e6a0806654abd81ca5661a2d019ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.519600 27290 tablet_service.cc:1467] Processing CreateTablet for tablet adc992deb712458aa793eaa11019de1f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:42.519820 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc992deb712458aa793eaa11019de1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.520567 28358 tablet_bootstrap.cc:492] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:42.521384 28358 tablet_bootstrap.cc:654] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.521754 27644 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:42.522624 27644 tablet_bootstrap.cc:654] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.523155 28793 tablet_bootstrap.cc:492] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:42.523155 28489 raft_consensus.cc:1050] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:42.523809 28489 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:42.523830 28393 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.524609 28791 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:42.525686 24764 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.526000 25034 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.526055 27176 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.526255 24897 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.526206 24896 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.526615 27747 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.528120 24485 catalog_manager.cc:5484] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.529295 27176 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.529652 27747 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.530109 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 01:58:42.530205 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 01:58:42.530205 27747 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.532577 28793 tablet_bootstrap.cc:654] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.532594 28358 tablet_bootstrap.cc:492] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:42.532665 28358 ts_tablet_manager.cc:1367] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240502 01:58:42.532876 27644 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:42.532950 27644 ts_tablet_manager.cc:1367] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:58:42.533087 28358 raft_consensus.cc:348] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.533185 28358 raft_consensus.cc:374] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.533214 28358 raft_consensus.cc:729] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.533322 28358 consensus_queue.cc:260] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.533365 27644 raft_consensus.cc:348] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.533458 27644 raft_consensus.cc:374] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.533491 27644 raft_consensus.cc:729] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.533578 28358 ts_tablet_manager.cc:1398] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.533603 27644 consensus_queue.cc:260] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.533852 27644 ts_tablet_manager.cc:1398] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.533804 24765 consensus_queue.cc:237] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:42.534016 27644 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:42.534833 27644 tablet_bootstrap.cc:654] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.536327 27177 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.536511 28393 raft_consensus.cc:1050] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:42.536684 25034 raft_consensus.cc:1261] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.536856 28393 consensus_queue.cc:237] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.537938 27941 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.538229 27941 consensus_queue.cc:1035] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.538596 24632 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.539855 24896 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.540092 28568 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.540197 27578 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.540685 28568 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.540869 28404 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.541013 28799 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.541280 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 2af8ec8342a9465e8af091c71c30599b
I20240502 01:58:42.541569 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 0345ae8459544e049edf0ef0bcc5136d
I20240502 01:58:42.541824 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet c7b1b5f0738c4e59a248ea7f946885a2
I20240502 01:58:42.541970 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af8ec8342a9465e8af091c71c30599b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.541990 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345ae8459544e049edf0ef0bcc5136d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.542269 27740 tablet_replica.cc:330] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.542393 27740 raft_consensus.cc:2229] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.542500 27740 raft_consensus.cc:2258] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.542497 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af8ec8342a9465e8af091c71c30599b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.542613 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af8ec8342a9465e8af091c71c30599b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.542640 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345ae8459544e049edf0ef0bcc5136d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.542762 28540 tablet_replica.cc:330] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.542784 27864 tablet_replica.cc:330] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.542876 28540 raft_consensus.cc:2229] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.542892 27864 raft_consensus.cc:2229] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.542948 27740 ts_tablet_manager.cc:1875] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.542984 28540 raft_consensus.cc:2258] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.543046 27864 raft_consensus.cc:2258] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.543428 28540 ts_tablet_manager.cc:1875] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.543490 27864 ts_tablet_manager.cc:1875] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.544035 28393 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.544807 27740 ts_tablet_manager.cc:1888] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:42.544883 27740 log.cc:1196] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/2af8ec8342a9465e8af091c71c30599b
I20240502 01:58:42.545158 27740 ts_tablet_manager.cc:1909] T 2af8ec8342a9465e8af091c71c30599b P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:42.545342 27864 ts_tablet_manager.cc:1888] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:42.545397 27864 log.cc:1196] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/2af8ec8342a9465e8af091c71c30599b
I20240502 01:58:42.545430 28540 ts_tablet_manager.cc:1888] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:42.545516 28540 log.cc:1196] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/2af8ec8342a9465e8af091c71c30599b
I20240502 01:58:42.545522 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26422e12ddf64e27b3913c60800e5a56 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.545651 27864 ts_tablet_manager.cc:1909] T 2af8ec8342a9465e8af091c71c30599b P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:42.545763 28540 ts_tablet_manager.cc:1909] T 2af8ec8342a9465e8af091c71c30599b P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:42.545818 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b1b5f0738c4e59a248ea7f946885a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.546008 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b1b5f0738c4e59a248ea7f946885a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.546033 28027 tablet_replica.cc:330] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.546144 28027 raft_consensus.cc:2229] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.546286 28027 raft_consensus.cc:2258] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.546305 28540 tablet_replica.cc:330] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.546413 28540 raft_consensus.cc:2229] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.546427 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 2af8ec8342a9465e8af091c71c30599b (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.546568 28540 raft_consensus.cc:2258] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.546649 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b1b5f0738c4e59a248ea7f946885a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.546908 28791 tablet_bootstrap.cc:654] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.546939 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 0345ae8459544e049edf0ef0bcc5136d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:42.546857 25034 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.547232 27177 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.548417 24486 catalog_manager.cc:5484] T 26422e12ddf64e27b3913c60800e5a56 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.548785 27644 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:42.548796 28365 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:42.548856 27644 ts_tablet_manager.cc:1367] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:58:42.548877 28365 ts_tablet_manager.cc:1367] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.001s
I20240502 01:58:42.549253 28365 raft_consensus.cc:348] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.549263 27644 raft_consensus.cc:348] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.549343 28365 raft_consensus.cc:374] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.549352 27644 raft_consensus.cc:374] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.549378 28365 raft_consensus.cc:729] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.549384 27644 raft_consensus.cc:729] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.549486 27644 consensus_queue.cc:260] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.549485 28365 consensus_queue.cc:260] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.549676 28814 tablet_replica.cc:330] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:42.549736 28365 ts_tablet_manager.cc:1398] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:42.549747 27644 ts_tablet_manager.cc:1398] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.549784 28814 raft_consensus.cc:2229] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.549822 28027 ts_tablet_manager.cc:1875] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.549888 28365 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:42.549891 28814 raft_consensus.cc:2258] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.551682 27740 tablet_replica.cc:330] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.551788 27740 raft_consensus.cc:2229] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.551821 28027 ts_tablet_manager.cc:1888] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:42.551872 28027 log.cc:1196] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/0345ae8459544e049edf0ef0bcc5136d
I20240502 01:58:42.551960 27740 raft_consensus.cc:2258] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.552098 28027 ts_tablet_manager.cc:1909] T 0345ae8459544e049edf0ef0bcc5136d P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:42.552320 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 2af8ec8342a9465e8af091c71c30599b (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.552541 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 26422e12ddf64e27b3913c60800e5a56 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.552832 24481 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.553534 28027 tablet_replica.cc:330] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.553651 28027 raft_consensus.cc:2229] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.553771 28027 raft_consensus.cc:2258] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.554288 24764 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.554811 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 0345ae8459544e049edf0ef0bcc5136d (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.554944 28791 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:42.555020 28791 ts_tablet_manager.cc:1367] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 01:58:42.555258 27740 ts_tablet_manager.cc:1875] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.555423 28791 raft_consensus.cc:348] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.556988 28791 raft_consensus.cc:374] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.557031 28791 raft_consensus.cc:729] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.557168 28791 consensus_queue.cc:260] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.557191 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 2af8ec8342a9465e8af091c71c30599b (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.557431 28540 ts_tablet_manager.cc:1875] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.557462 27864 tablet_replica.cc:330] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.557592 27864 raft_consensus.cc:2229] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.557693 27864 raft_consensus.cc:2258] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.558122 27864 ts_tablet_manager.cc:1875] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.558426 28791 ts_tablet_manager.cc:1398] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:42.558585 28791 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:42.558765 24897 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.559182 28540 ts_tablet_manager.cc:1888] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:42.559239 28540 log.cc:1196] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/c7b1b5f0738c4e59a248ea7f946885a2
I20240502 01:58:42.559460 28540 ts_tablet_manager.cc:1909] T c7b1b5f0738c4e59a248ea7f946885a2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:42.559533 28365 tablet_bootstrap.cc:654] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.559537 28791 tablet_bootstrap.cc:654] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:42.560173 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet c7b1b5f0738c4e59a248ea7f946885a2 (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.560438 27177 raft_consensus.cc:2932] T 26422e12ddf64e27b3913c60800e5a56 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.561733 27941 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.562209 28027 ts_tablet_manager.cc:1875] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.564105 27578 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.564294 24896 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.565091 27740 ts_tablet_manager.cc:1888] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:42.565147 27740 log.cc:1196] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/0345ae8459544e049edf0ef0bcc5136d
I20240502 01:58:42.565392 27740 ts_tablet_manager.cc:1909] T 0345ae8459544e049edf0ef0bcc5136d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:42.566022 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 0345ae8459544e049edf0ef0bcc5136d (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.566164 28027 ts_tablet_manager.cc:1888] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.566218 28027 log.cc:1196] T 26422e12ddf64e27b3913c60800e5a56 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/26422e12ddf64e27b3913c60800e5a56
I20240502 01:58:42.566215 24630 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:42.566594 27740 tablet_replica.cc:330] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.565624 27864 ts_tablet_manager.cc:1888] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:42.566874 27864 log.cc:1196] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7b1b5f0738c4e59a248ea7f946885a2
I20240502 01:58:42.566780 27740 raft_consensus.cc:2229] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.567088 27740 raft_consensus.cc:2258] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.567107 27864 ts_tablet_manager.cc:1909] T c7b1b5f0738c4e59a248ea7f946885a2 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:42.567620 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet c7b1b5f0738c4e59a248ea7f946885a2 (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.567852 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 26422e12ddf64e27b3913c60800e5a56 (table rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]) successfully deleted
I20240502 01:58:42.561380 28791 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:42.568285 28791 ts_tablet_manager.cc:1367] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:42.568665 28791 raft_consensus.cc:348] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.568755 28791 raft_consensus.cc:374] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.568787 28791 raft_consensus.cc:729] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.569051 24765 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.569186 27740 ts_tablet_manager.cc:1875] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.569332 24486 catalog_manager.cc:5484] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.569537 27923 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.569839 28791 consensus_queue.cc:260] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.570444 28791 ts_tablet_manager.cc:1398] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:42.571230 27740 ts_tablet_manager.cc:1888] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:42.571296 27740 log.cc:1196] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7b1b5f0738c4e59a248ea7f946885a2
I20240502 01:58:42.571532 27740 ts_tablet_manager.cc:1909] T c7b1b5f0738c4e59a248ea7f946885a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:42.572075 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet c7b1b5f0738c4e59a248ea7f946885a2 (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.572810 28365 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:42.572901 28365 ts_tablet_manager.cc:1367] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
I20240502 01:58:42.572983 28793 tablet_bootstrap.cc:492] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:42.573055 28793 ts_tablet_manager.cc:1367] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20240502 01:58:42.573340 28365 raft_consensus.cc:348] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.573429 28365 raft_consensus.cc:374] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.573459 28365 raft_consensus.cc:729] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.573562 28365 consensus_queue.cc:260] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.573796 28365 ts_tablet_manager.cc:1398] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:42.576419 24764 consensus_queue.cc:237] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:42.577044 27177 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.577131 28793 raft_consensus.cc:348] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.577272 28793 raft_consensus.cc:374] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:42.577350 28793 raft_consensus.cc:729] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.577590 27578 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.577657 27923 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.577775 28793 consensus_queue.cc:260] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.578030 28568 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.578168 24896 raft_consensus.cc:1261] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.579305 28568 consensus_queue.cc:1035] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.579514 28793 ts_tablet_manager.cc:1398] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:42.579994 27747 raft_consensus.cc:1050] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:42.580343 27747 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.581521 27747 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.581792 24897 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.581858 27445 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.582336 27923 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.582732 28816 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.583454 27177 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.583982 25034 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.584167 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e720d8363bb049938798ab0b9aabc28d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.584707 28489 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.584918 28568 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.585403 24486 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.585855 24896 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.586318 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a383190824e4611a9e279b05bfe4e2a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.587702 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet e720d8363bb049938798ab0b9aabc28d with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:42.587864 28027 tablet_replica.cc:330] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:42.587962 28027 raft_consensus.cc:2229] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.588092 28027 raft_consensus.cc:2258] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.588224 24765 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:42.588415 24486 catalog_manager.cc:5484] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.590618 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a383190824e4611a9e279b05bfe4e2a with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:42.591045 28540 tablet_replica.cc:330] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:42.591235 28540 raft_consensus.cc:2229] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.591406 28540 pending_rounds.cc:62] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Trying to abort 1 pending ops.
I20240502 01:58:42.591486 28540 pending_rounds.cc:68] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023063541910089728 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1a383190824e4611a9e279b05bfe4e2a" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } } }
I20240502 01:58:42.591647 28540 raft_consensus.cc:2894] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:58:42.591713 28540 raft_consensus.cc:2866] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:58:42.591768 28540 raft_consensus.cc:2258] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.592588 28489 raft_consensus.cc:1050] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:42.592933 28489 consensus_queue.cc:237] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:42.594285 27312 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.594285 25034 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.594760 27747 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.594789 28027 ts_tablet_manager.cc:1875] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.594951 28816 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.595583 24765 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.596295 28816 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.596482 28027 ts_tablet_manager.cc:1888] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.596551 28027 log.cc:1196] T e720d8363bb049938798ab0b9aabc28d P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/e720d8363bb049938798ab0b9aabc28d
I20240502 01:58:42.597019 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet e720d8363bb049938798ab0b9aabc28d (table rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]) successfully deleted
I20240502 01:58:42.598115 28814 ts_tablet_manager.cc:1875] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.598693 27578 raft_consensus.cc:2932] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:42.599812 27747 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.601729 24487 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:42.602138 27312 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.603217 28814 ts_tablet_manager.cc:1888] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:42.603323 28814 log.cc:1196] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/0345ae8459544e049edf0ef0bcc5136d
I20240502 01:58:42.604215 24481 catalog_manager.cc:5484] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.604691 25034 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.604902 24765 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.606062 27177 raft_consensus.cc:1261] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:42.606516 27865 consensus_queue.cc:1035] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.608215 24630 consensus_queue.cc:237] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
W20240502 01:58:42.609476 28819 consensus_queue.cc:1159] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "588bfc090a5247598cd5d3cee5ad2007" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 01:58:42.610144 28540 ts_tablet_manager.cc:1875] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.611279 25034 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.611313 27311 raft_consensus.cc:1261] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.611313 27312 raft_consensus.cc:1261] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:42.611704 28799 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.611963 27865 consensus_queue.cc:1035] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:42.612002 28799 consensus_queue.cc:1035] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.612085 28540 ts_tablet_manager.cc:1888] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.612134 28540 log.cc:1196] T 1a383190824e4611a9e279b05bfe4e2a P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a383190824e4611a9e279b05bfe4e2a
I20240502 01:58:42.612216 24486 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]). Expected version 1 got 0
I20240502 01:58:42.613464 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 1a383190824e4611a9e279b05bfe4e2a (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:42.614768 24481 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]). Expected version 1 got 0
I20240502 01:58:42.615286 28799 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.615895 28814 ts_tablet_manager.cc:1909] T 0345ae8459544e049edf0ef0bcc5136d P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:42.616143 28489 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.617225 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 0345ae8459544e049edf0ef0bcc5136d (table rebalance_test_table_10 [id=7787f11e7d3d4b3a9764410b89032b28]) successfully deleted
I20240502 01:58:42.617766 24896 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.618728 24481 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.619233 28606 mvcc.cc:205] Tried to move back new op lower bound from 7023063542189871104 to 7023063541695639552. Current Snapshot: MvccSnapshot[applied={T|T < 7023063542189871104}]
I20240502 01:58:42.620501 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a094659d539e4c4f9861e356168c5894 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.620726 28598 mvcc.cc:205] Tried to move back new op lower bound from 7023063542189871104 to 7023063541695639552. Current Snapshot: MvccSnapshot[applied={T|T < 7023063542189871104}]
I20240502 01:58:42.620707 25032 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.622592 27445 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.623286 24485 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]). Expected version 1 got 0
I20240502 01:58:42.622924 24486 catalog_manager.cc:5484] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.623718 27311 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.624318 24481 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]). Expected version 1 got 0
I20240502 01:58:42.625490 28842 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:42.625766 28842 tablet.cc:1620] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:42.626005 28841 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:42.626752 24632 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.627907 24896 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.627996 28841 tablet.cc:1620] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:42.628170 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet a094659d539e4c4f9861e356168c5894 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.628371 27864 tablet_replica.cc:330] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.628502 27864 raft_consensus.cc:2229] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.628500 27747 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.628639 27864 raft_consensus.cc:2258] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.630435 27445 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.630398 28799 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.631860 27923 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.631821 24896 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.631948 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69182e5f4ae7401f89d22f063fa8df63 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.633515 27445 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.634665 24483 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.635089 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6] alter complete (version 1)
I20240502 01:58:42.636310 28447 raft_consensus.cc:482] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:42.636387 28447 raft_consensus.cc:504] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.636513 27864 ts_tablet_manager.cc:1875] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.636778 28447 leader_election.cc:288] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:42.637140 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc992deb712458aa793eaa11019de1f" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:42.637243 27311 raft_consensus.cc:2454] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:42.637605 27377 leader_election.cc:302] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:42.637755 28447 raft_consensus.cc:2781] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:42.637802 28447 raft_consensus.cc:482] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:42.637837 28447 raft_consensus.cc:3037] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.638640 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 69182e5f4ae7401f89d22f063fa8df63 with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.638702 28447 raft_consensus.cc:504] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.638823 27740 tablet_replica.cc:330] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.638948 27740 raft_consensus.cc:2229] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.639037 28447 leader_election.cc:288] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:42.639111 27740 raft_consensus.cc:2258] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.639767 27740 ts_tablet_manager.cc:1875] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.639967 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc992deb712458aa793eaa11019de1f" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:42.640061 27311 raft_consensus.cc:3037] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.640717 27311 raft_consensus.cc:2454] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:42.640960 27377 leader_election.cc:302] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:42.641247 28447 raft_consensus.cc:2781] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:42.641319 28447 raft_consensus.cc:686] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:42.641424 28447 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:42.641638 27740 ts_tablet_manager.cc:1888] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.641687 27740 log.cc:1196] T 69182e5f4ae7401f89d22f063fa8df63 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/69182e5f4ae7401f89d22f063fa8df63
I20240502 01:58:42.642699 27311 raft_consensus.cc:1203] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:42.643015 24486 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.644726 27864 ts_tablet_manager.cc:1888] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.644800 27864 log.cc:1196] T a094659d539e4c4f9861e356168c5894 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/a094659d539e4c4f9861e356168c5894
I20240502 01:58:42.645313 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet a094659d539e4c4f9861e356168c5894 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:42.645325 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 69182e5f4ae7401f89d22f063fa8df63 (table rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]) successfully deleted
I20240502 01:58:42.646551 25034 raft_consensus.cc:2932] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.652554 24486 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6]). Expected version 1 got 0
I20240502 01:58:42.653784 28851 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:42.654160 28851 tablet.cc:1620] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:42.657886 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc992deb712458aa793eaa11019de1f" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:42.657964 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc992deb712458aa793eaa11019de1f" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:42.658035 27578 raft_consensus.cc:2454] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:42.662560 24765 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:42.663852 25034 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:42.664012 24700 consensus_peers.cc:485] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63312ec15b034e47bc2dd484b57d4648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.664376 24896 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.664350 27941 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.664958 27941 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.667061 28404 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:42.667449 25034 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:42.669488 24481 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.670192 24896 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:42.672534 28826 raft_consensus.cc:1050] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:42.672976 28826 consensus_queue.cc:237] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.674614 27445 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.674785 25034 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.674953 24897 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.675166 27923 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.675869 27747 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.676179 27747 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.678026 27747 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.678550 27445 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.679950 25034 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.680382 24487 catalog_manager.cc:5484] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.680553 27578 consensus_queue.cc:237] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:42.683440 27177 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:42.683861 27510 consensus_peers.cc:485] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be -> Peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): Couldn't send request to peer e0b9e7df4e8543fa9b8e3ffc24330657. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 823fe241dc1b40cca121a22d669e4226. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.684334 28872 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:42.684563 24897 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.686055 24632 consensus_queue.cc:237] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.686259 27867 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:42.686669 27177 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:42.687207 24632 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:42.688345 24897 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.688551 27445 raft_consensus.cc:1261] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.688724 27865 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:42.689393 28816 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.689533 24481 catalog_manager.cc:5484] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:42.689671 28816 consensus_queue.cc:1035] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.690601 24632 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:42.693595 28816 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.695012 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 752d360cfdd64846b6cd37c9b859d7a8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.695362 27445 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.695914 24897 raft_consensus.cc:2932] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.697130 24486 catalog_manager.cc:5484] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.699159 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.699828 27740 tablet_replica.cc:330] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.699967 27740 raft_consensus.cc:2229] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.700106 27740 raft_consensus.cc:2258] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.700747 27740 ts_tablet_manager.cc:1875] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.702533 24765 consensus_queue.cc:237] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.702709 27740 ts_tablet_manager.cc:1888] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.702786 27740 log.cc:1196] T 752d360cfdd64846b6cd37c9b859d7a8 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/752d360cfdd64846b6cd37c9b859d7a8
I20240502 01:58:42.703305 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 752d360cfdd64846b6cd37c9b859d7a8 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
W20240502 01:58:42.704075 24701 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebe8db639839433d967583371351eacb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.704138 25034 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.704456 24897 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.704509 27941 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.704809 28393 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.706588 28568 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.707103 25034 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.708349 24897 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:42.709074 24481 catalog_manager.cc:5484] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.717665 28826 raft_consensus.cc:1050] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:42.717794 25034 consensus_queue.cc:237] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:42.718089 28826 consensus_queue.cc:237] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:42.721014 24632 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:42.721328 27312 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.721755 27177 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:42.721705 27923 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.722121 28534 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:42.722211 24765 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.722326 24897 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:42.722487 28534 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:42.722666 28816 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.722985 28816 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:42.724974 27923 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.725468 27894 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:42.725947 24632 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:42.726981 27312 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.727363 27177 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:42.727511 24487 catalog_manager.cc:5484] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.728415 28872 raft_consensus.cc:482] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:42.728499 28872 raft_consensus.cc:504] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.728888 28872 leader_election.cc:288] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:42.728988 24486 catalog_manager.cc:5484] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.731865 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584cbed9fc494fc2b32c1bfd2571d88b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:42.731981 24632 raft_consensus.cc:2454] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:42.732587 27513 leader_election.cc:302] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
W20240502 01:58:42.732627 24966 consensus_peers.cc:485] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6560bc797c024a90aa87515d0ef2c4c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.732856 28872 raft_consensus.cc:2781] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:42.732919 28872 raft_consensus.cc:482] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:42.732990 28872 raft_consensus.cc:3037] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.732954 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584cbed9fc494fc2b32c1bfd2571d88b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:42.733063 27177 raft_consensus.cc:2454] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:42.733966 28872 raft_consensus.cc:504] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.734310 28872 leader_election.cc:288] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:42.735049 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584cbed9fc494fc2b32c1bfd2571d88b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:42.735150 27177 raft_consensus.cc:3037] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.735170 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584cbed9fc494fc2b32c1bfd2571d88b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:42.735266 24630 raft_consensus.cc:3037] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.735303 24632 consensus_queue.cc:237] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:42.735240 24897 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.735945 27177 raft_consensus.cc:2454] T 584cbed9fc494fc2b32c1bfd2571d88b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:42.736135 24630 raft_consensus.cc:2454] T 584cbed9fc494fc2b32c1bfd2571d88b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:42.736728 27513 leader_election.cc:302] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:42.736800 27312 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.736892 24765 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.738691 28816 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.739702 24897 raft_consensus.cc:1261] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.740280 28816 consensus_queue.cc:1035] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.740806 28872 raft_consensus.cc:2781] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:42.740890 28872 raft_consensus.cc:686] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:42.740873 28816 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.741009 28872 consensus_queue.cc:237] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:42.742476 24486 catalog_manager.cc:5484] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.743005 27312 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.743173 24485 catalog_manager.cc:5484] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.742511 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:42.744423 24897 raft_consensus.cc:2932] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:42.744735 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ede6fc65d044804848e6f61e8c238c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.748642 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:42.748842 27864 tablet_replica.cc:330] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:42.748971 27864 raft_consensus.cc:2229] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.749128 27864 raft_consensus.cc:2258] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.749730 27864 ts_tablet_manager.cc:1875] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.749820 28841 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:42.750103 28841 tablet.cc:1620] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:42.751827 27864 ts_tablet_manager.cc:1888] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.751897 27864 log.cc:1196] T 9ede6fc65d044804848e6f61e8c238c0 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/9ede6fc65d044804848e6f61e8c238c0
I20240502 01:58:42.752806 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 9ede6fc65d044804848e6f61e8c238c0 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:42.752862 28842 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:42.753167 28842 tablet.cc:1620] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:42.755290 28851 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:42.755573 28851 tablet.cc:1620] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:42.757674 24487 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1261bfec795a4dd8bd9052846a658cd6] alter complete (version 2)
I20240502 01:58:42.761656 28903 ts_tablet_manager.cc:902] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:42.762183 28903 tablet_copy_client.cc:308] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:42.762611 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet d5ec1158a7414039b6f4bc855207c938 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:55633)
I20240502 01:58:42.762701 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet d5ec1158a7414039b6f4bc855207c938 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:55633: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-d5ec1158a7414039b6f4bc855207c938
I20240502 01:58:42.763392 24918 tablet_copy_source_session.cc:205] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:42.764142 28903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ec1158a7414039b6f4bc855207c938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.780169 28903 tablet_copy_client.cc:783] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:42.780335 28903 tablet_copy_client.cc:647] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:42.781476 28903 tablet_copy_client.cc:523] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:42.782877 28903 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:42.782971 28930 ts_tablet_manager.cc:902] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:42.783448 28930 tablet_copy_client.cc:308] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:42.784132 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 488816b0d5d542bcb5fa12ff7e70fd40 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:45689)
I20240502 01:58:42.784215 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 488816b0d5d542bcb5fa12ff7e70fd40 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:45689: session id = bb653087e8554c7591cf547dd969f3be-488816b0d5d542bcb5fa12ff7e70fd40
I20240502 01:58:42.785351 25054 tablet_copy_source_session.cc:205] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:42.786314 28930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488816b0d5d542bcb5fa12ff7e70fd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.793676 28930 tablet_copy_client.cc:783] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:42.794565 28941 ts_tablet_manager.cc:902] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:42.795099 28941 tablet_copy_client.cc:308] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:42.793982 28930 tablet_copy_client.cc:647] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:42.795509 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 9f999b09b79a4632936833f351c4b28b from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:47383)
I20240502 01:58:42.795709 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 9f999b09b79a4632936833f351c4b28b from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:47383: session id = 9dd619a595e94004ac0f7f5a9054e7cc-9f999b09b79a4632936833f351c4b28b
I20240502 01:58:42.796550 28930 tablet_copy_client.cc:523] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:42.796672 24785 tablet_copy_source_session.cc:205] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:42.797492 28941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f999b09b79a4632936833f351c4b28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.797827 28930 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:42.799618 28941 tablet_copy_client.cc:783] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:42.799772 28941 tablet_copy_client.cc:647] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:42.800844 28941 tablet_copy_client.cc:523] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:42.802399 28941 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:42.804822 24897 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:42.806921 27445 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.807474 24630 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:42.807535 28708 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.808579 28708 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:42.808961 28396 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.810842 24630 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.811012 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b3edec06b3d8450a8cc3002be70c848a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:42.811471 27445 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:42.813369 24484 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:42.818773 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet b3edec06b3d8450a8cc3002be70c848a with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:42.819017 27740 tablet_replica.cc:330] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:42.819161 27740 raft_consensus.cc:2229] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.819316 27740 raft_consensus.cc:2258] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.820010 27740 ts_tablet_manager.cc:1875] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:42.821812 28903 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:42.822019 27740 ts_tablet_manager.cc:1888] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:42.822077 27740 log.cc:1196] T b3edec06b3d8450a8cc3002be70c848a P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3edec06b3d8450a8cc3002be70c848a
I20240502 01:58:42.822108 28903 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:42.822273 28903 ts_tablet_manager.cc:1367] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.039s user 0.030s sys 0.002s
I20240502 01:58:42.822510 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet b3edec06b3d8450a8cc3002be70c848a (table rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]) successfully deleted
I20240502 01:58:42.822728 28903 raft_consensus.cc:348] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:42.822957 28903 raft_consensus.cc:729] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:42.823129 28903 consensus_queue.cc:260] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:42.825452 28903 ts_tablet_manager.cc:1398] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:42.827513 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-d5ec1158a7414039b6f4bc855207c938 received from {username='slave'} at 127.17.243.7:55633
I20240502 01:58:42.827589 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-d5ec1158a7414039b6f4bc855207c938 on tablet d5ec1158a7414039b6f4bc855207c938 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:42.829866 28930 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:42.830135 28930 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:42.830318 28930 ts_tablet_manager.cc:1367] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.001s
I20240502 01:58:42.830665 28930 raft_consensus.cc:348] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.830761 28930 raft_consensus.cc:729] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:42.830853 28930 consensus_queue.cc:260] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.831092 28930 ts_tablet_manager.cc:1398] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:42.831530 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-488816b0d5d542bcb5fa12ff7e70fd40 received from {username='slave'} at 127.17.243.8:45689
I20240502 01:58:42.831597 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session bb653087e8554c7591cf547dd969f3be-488816b0d5d542bcb5fa12ff7e70fd40 on tablet 488816b0d5d542bcb5fa12ff7e70fd40 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:42.840888 28941 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:42.841179 28941 tablet_bootstrap.cc:492] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:42.841418 28941 ts_tablet_manager.cc:1367] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.039s user 0.027s sys 0.000s
I20240502 01:58:42.841830 28941 raft_consensus.cc:348] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:42.841986 28941 raft_consensus.cc:729] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:42.842119 28941 consensus_queue.cc:260] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:42.842438 28941 ts_tablet_manager.cc:1398] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:42.843271 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-9f999b09b79a4632936833f351c4b28b received from {username='slave'} at 127.17.243.5:47383
I20240502 01:58:42.843356 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-9f999b09b79a4632936833f351c4b28b on tablet 9f999b09b79a4632936833f351c4b28b with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:42.860091 28315 raft_consensus.cc:482] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:42.860183 28315 raft_consensus.cc:504] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.860355 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:42.860567 28315 leader_election.cc:288] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:42.861063 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e6a0806654abd81ca5661a2d019ce" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:42.861207 27312 raft_consensus.cc:2454] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:42.861387 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e6a0806654abd81ca5661a2d019ce" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:42.861517 27578 raft_consensus.cc:2454] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:42.861572 27377 leader_election.cc:302] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:42.861743 28315 raft_consensus.cc:2781] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:42.861796 28315 raft_consensus.cc:482] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:42.861835 28315 raft_consensus.cc:3037] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.862754 28315 raft_consensus.cc:504] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.863102 28315 leader_election.cc:288] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:42.863643 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e6a0806654abd81ca5661a2d019ce" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:42.863735 27578 raft_consensus.cc:3037] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.864637 27578 raft_consensus.cc:2454] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:42.865065 27380 leader_election.cc:302] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:42.865432 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85e6a0806654abd81ca5661a2d019ce" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:42.865540 27311 raft_consensus.cc:3037] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:42.866004 28841 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:42.866333 28842 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:42.866649 28851 tablet.cc:1722] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:42.870250 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=1261bfec795a4dd8bd9052846a658cd6] alter complete (version 3)
I20240502 01:58:42.870484 27311 raft_consensus.cc:2454] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:42.871122 28315 raft_consensus.cc:2781] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:42.871232 28315 raft_consensus.cc:686] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:42.871345 28315 consensus_queue.cc:237] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:42.873023 24487 catalog_manager.cc:5484] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:42.891824 25034 consensus_queue.cc:237] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:42.893297 24630 raft_consensus.cc:1261] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.893308 24765 raft_consensus.cc:1261] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:42.893713 28534 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:42.893959 24963 consensus_peers.cc:485] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 693e0959e61c44c59f797e4060187cef. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:42.894167 28534 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:42.895594 27017 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:42.896072 24630 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:42.896272 24765 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:42.898376 24487 catalog_manager.cc:5484] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:42.939981 28930 ts_tablet_manager.cc:902] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:42.940404 28930 tablet_copy_client.cc:308] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:42.941213 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 4ac6a4310330499483182c93827343c0 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:39965)
I20240502 01:58:42.941306 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 4ac6a4310330499483182c93827343c0 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:39965: session id = bb653087e8554c7591cf547dd969f3be-4ac6a4310330499483182c93827343c0
I20240502 01:58:42.942318 24785 tablet_copy_source_session.cc:205] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:42.943116 28930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac6a4310330499483182c93827343c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:42.944701 28930 tablet_copy_client.cc:783] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:42.944842 28930 tablet_copy_client.cc:647] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:42.946249 28930 tablet_copy_client.cc:523] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:42.947432 28930 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:42.972245 24485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:42.972347 24485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:42.975438 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet c95fa6f5402d42d09a68f3793b1f000f
I20240502 01:58:42.976125 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet c95fa6f5402d42d09a68f3793b1f000f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:42.976125 27555 tablet_service.cc:1514] Processing DeleteTablet for tablet c95fa6f5402d42d09a68f3793b1f000f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:42.976711 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet c95fa6f5402d42d09a68f3793b1f000f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:42 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:42.976900 28814 tablet_replica.cc:330] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:42.977027 28814 raft_consensus.cc:2229] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.977137 28814 raft_consensus.cc:2258] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.977844 28814 ts_tablet_manager.cc:1875] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.979959 28814 ts_tablet_manager.cc:1888] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.980027 28814 log.cc:1196] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/c95fa6f5402d42d09a68f3793b1f000f
I20240502 01:58:42.980212 28930 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:42.980331 28814 ts_tablet_manager.cc:1909] T c95fa6f5402d42d09a68f3793b1f000f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:42.980444 28930 tablet_bootstrap.cc:492] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:42.980587 28930 ts_tablet_manager.cc:1367] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.000s
I20240502 01:58:42.981002 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table_new_name [id=1261bfec795a4dd8bd9052846a658cd6]) successfully deleted
I20240502 01:58:42.980999 28930 raft_consensus.cc:348] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.981117 28930 raft_consensus.cc:729] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:42.981220 28930 consensus_queue.cc:260] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:42.981482 29049 tablet_replica.cc:330] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:42.981772 29049 raft_consensus.cc:2229] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:42.982018 29049 raft_consensus.cc:2258] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.982668 29049 ts_tablet_manager.cc:1875] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.982884 29050 tablet_replica.cc:330] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:42.984470 29050 raft_consensus.cc:2229] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:42.984948 28930 ts_tablet_manager.cc:1398] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:42.985114 29050 raft_consensus.cc:2258] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:42.985678 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-4ac6a4310330499483182c93827343c0 received from {username='slave'} at 127.17.243.8:39965
I20240502 01:58:42.985770 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session bb653087e8554c7591cf547dd969f3be-4ac6a4310330499483182c93827343c0 on tablet 4ac6a4310330499483182c93827343c0 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:42.988498 29050 ts_tablet_manager.cc:1875] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:42.989481 29049 ts_tablet_manager.cc:1888] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.989567 29049 log.cc:1196] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/c95fa6f5402d42d09a68f3793b1f000f
I20240502 01:58:42.989843 29049 ts_tablet_manager.cc:1909] T c95fa6f5402d42d09a68f3793b1f000f P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:42.991161 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table_new_name [id=1261bfec795a4dd8bd9052846a658cd6]) successfully deleted
I20240502 01:58:42.991870 29050 ts_tablet_manager.cc:1888] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:42.991999 29050 log.cc:1196] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/c95fa6f5402d42d09a68f3793b1f000f
I20240502 01:58:42.992357 29050 ts_tablet_manager.cc:1909] T c95fa6f5402d42d09a68f3793b1f000f P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:42.993125 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet c95fa6f5402d42d09a68f3793b1f000f (table rebalancer_dynamic_table_new_name [id=1261bfec795a4dd8bd9052846a658cd6]) successfully deleted
I20240502 01:58:43.000524 27290 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:43.006395 24610 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:43.012912 27424 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.014547 24743 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:58:43.014925 27155 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.019089 25012 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:58:43.027787 27555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.036577 24877 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:58:43.039638 28941 ts_tablet_manager.cc:902] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:43.040066 28941 tablet_copy_client.cc:308] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:43.043552 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 01:58:43.043666 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 01:58:43.046135 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 9ede6fc65d044804848e6f61e8c238c0
I20240502 01:58:43.046211 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 63312ec15b034e47bc2dd484b57d4648 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:47383)
I20240502 01:58:43.046283 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 63312ec15b034e47bc2dd484b57d4648 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:47383: session id = 9dd619a595e94004ac0f7f5a9054e7cc-63312ec15b034e47bc2dd484b57d4648
I20240502 01:58:43.046474 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet 9f999b09b79a4632936833f351c4b28b
I20240502 01:58:43.046722 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet fa3e57ee511e4d9886fcfed3851e4c38
I20240502 01:58:43.046851 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:43.046851 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:43.047042 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ede6fc65d044804848e6f61e8c238c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:43.047187 24785 tablet_copy_source_session.cc:205] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.047232 29049 tablet_replica.cc:330] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:43.047362 29049 raft_consensus.cc:2229] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.047502 28540 tablet_replica.cc:330] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:43.047506 29049 raft_consensus.cc:2258] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.047647 28540 raft_consensus.cc:2229] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.047832 28540 raft_consensus.cc:2258] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.047915 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet fa3e57ee511e4d9886fcfed3851e4c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:43.047988 28027 tablet_replica.cc:330] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:43.048024 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f999b09b79a4632936833f351c4b28b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:43.048105 28027 raft_consensus.cc:2229] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.048142 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f999b09b79a4632936833f351c4b28b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:43.048246 28027 raft_consensus.cc:2258] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.048298 27740 tablet_replica.cc:330] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:43.048323 27864 tablet_replica.cc:330] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:43.048434 27864 raft_consensus.cc:2229] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.048435 27740 raft_consensus.cc:2229] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.048581 27740 raft_consensus.cc:2258] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.048625 27864 raft_consensus.cc:2258] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.047514 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f999b09b79a4632936833f351c4b28b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:43.048143 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet fa3e57ee511e4d9886fcfed3851e4c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:43.049376 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f999b09b79a4632936833f351c4b28b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:43.049517 28814 tablet_replica.cc:330] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:43.049508 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet fa3e57ee511e4d9886fcfed3851e4c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:43.049612 28814 raft_consensus.cc:2229] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:58:43.049691 28814 raft_consensus.cc:2258] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:58:43.050294 28814 ts_tablet_manager.cc:1875] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.050294 29049 ts_tablet_manager.cc:1875] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.050726 28027 ts_tablet_manager.cc:1875] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.052304 29049 ts_tablet_manager.cc:1888] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:43.052304 28814 ts_tablet_manager.cc:1888] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.052366 29049 log.cc:1196] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/9ede6fc65d044804848e6f61e8c238c0
I20240502 01:58:43.052373 28814 log.cc:1196] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/9f999b09b79a4632936833f351c4b28b
I20240502 01:58:43.052626 28814 ts_tablet_manager.cc:1909] T 9f999b09b79a4632936833f351c4b28b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:43.052626 29049 ts_tablet_manager.cc:1909] T 9ede6fc65d044804848e6f61e8c238c0 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:43.052758 28027 ts_tablet_manager.cc:1888] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:43.052814 28027 log.cc:1196] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/9ede6fc65d044804848e6f61e8c238c0
I20240502 01:58:43.053073 28027 ts_tablet_manager.cc:1909] T 9ede6fc65d044804848e6f61e8c238c0 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:43.053328 27740 ts_tablet_manager.cc:1875] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.053786 28540 ts_tablet_manager.cc:1875] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.054019 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 9ede6fc65d044804848e6f61e8c238c0 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.054236 28027 tablet_replica.cc:330] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:43.054342 28027 raft_consensus.cc:2229] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.054461 28027 raft_consensus.cc:2258] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.054947 28027 ts_tablet_manager.cc:1875] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.055435 27740 ts_tablet_manager.cc:1888] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.055495 27740 log.cc:1196] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/9f999b09b79a4632936833f351c4b28b
I20240502 01:58:43.055645 28540 ts_tablet_manager.cc:1888] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:43.055693 28540 log.cc:1196] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ede6fc65d044804848e6f61e8c238c0
I20240502 01:58:43.055732 27740 ts_tablet_manager.cc:1909] T 9f999b09b79a4632936833f351c4b28b P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:43.055905 28540 ts_tablet_manager.cc:1909] T 9ede6fc65d044804848e6f61e8c238c0 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:43.056171 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 9f999b09b79a4632936833f351c4b28b (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.056362 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 9f999b09b79a4632936833f351c4b28b (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.056654 28941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63312ec15b034e47bc2dd484b57d4648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.056962 28027 ts_tablet_manager.cc:1888] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:43.057019 28027 log.cc:1196] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa3e57ee511e4d9886fcfed3851e4c38
I20240502 01:58:43.057148 27864 ts_tablet_manager.cc:1875] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.057278 28027 ts_tablet_manager.cc:1909] T fa3e57ee511e4d9886fcfed3851e4c38 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:43.057374 28540 tablet_replica.cc:330] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:43.057485 28540 raft_consensus.cc:2229] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.058734 28941 tablet_copy_client.cc:783] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.058864 28941 tablet_copy_client.cc:647] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.059160 27864 ts_tablet_manager.cc:1888] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.059227 27864 log.cc:1196] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f999b09b79a4632936833f351c4b28b
I20240502 01:58:43.059480 27864 ts_tablet_manager.cc:1909] T 9f999b09b79a4632936833f351c4b28b P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:43.059739 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet fa3e57ee511e4d9886fcfed3851e4c38 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.059871 28779 raft_consensus.cc:1050] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:43.059921 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 9ede6fc65d044804848e6f61e8c238c0 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.060042 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 9f999b09b79a4632936833f351c4b28b (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.060268 28779 consensus_queue.cc:237] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:43.057585 28540 raft_consensus.cc:2258] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.060660 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 9ede6fc65d044804848e6f61e8c238c0 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.060904 27740 tablet_replica.cc:330] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:43.061066 27740 raft_consensus.cc:2229] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.061235 27740 raft_consensus.cc:2258] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.061496 28941 tablet_copy_client.cc:523] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.061480 28540 ts_tablet_manager.cc:1875] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.061656 27740 ts_tablet_manager.cc:1875] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.062007 25032 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.062428 27311 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.062711 28941 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:43.063515 24765 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.063520 27740 ts_tablet_manager.cc:1888] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:43.063498 27093 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.063581 27740 log.cc:1196] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/fa3e57ee511e4d9886fcfed3851e4c38
I20240502 01:58:43.063795 27740 ts_tablet_manager.cc:1909] T fa3e57ee511e4d9886fcfed3851e4c38 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:43.063795 27093 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.064388 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet fa3e57ee511e4d9886fcfed3851e4c38 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.064657 27645 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.064824 28540 ts_tablet_manager.cc:1888] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:43.064927 28540 log.cc:1196] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa3e57ee511e4d9886fcfed3851e4c38
I20240502 01:58:43.065171 28540 ts_tablet_manager.cc:1909] T fa3e57ee511e4d9886fcfed3851e4c38 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:43.065901 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet fa3e57ee511e4d9886fcfed3851e4c38 (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.066179 28540 tablet_replica.cc:330] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:43.066327 28540 raft_consensus.cc:2229] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.066489 28540 raft_consensus.cc:2258] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.067129 28540 ts_tablet_manager.cc:1875] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.070433 28540 ts_tablet_manager.cc:1888] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.070503 28540 log.cc:1196] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f999b09b79a4632936833f351c4b28b
I20240502 01:58:43.072422 28540 ts_tablet_manager.cc:1909] T 9f999b09b79a4632936833f351c4b28b P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:43.073031 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 9f999b09b79a4632936833f351c4b28b (table rebalance_test_table_12 [id=534ca96be10f4f9c82a35c4df0620f8a]) successfully deleted
I20240502 01:58:43.074229 27645 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.077423 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:43.077651 24479 catalog_manager.cc:5484] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:43.078428 25032 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.081595 27311 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.082715 24897 consensus_queue.cc:237] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:43.085536 27312 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:43.086822 28708 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:43.087503 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 51462d8e7b414a169f3cfedb6b7ca585 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d]), partition=
I20240502 01:58:43.087734 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51462d8e7b414a169f3cfedb6b7ca585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.087791 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 51462d8e7b414a169f3cfedb6b7ca585 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d]), partition=
I20240502 01:58:43.087986 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 51462d8e7b414a169f3cfedb6b7ca585 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d]), partition=
I20240502 01:58:43.088016 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51462d8e7b414a169f3cfedb6b7ca585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.088178 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51462d8e7b414a169f3cfedb6b7ca585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.090531 25032 raft_consensus.cc:1261] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:43.091212 28708 consensus_queue.cc:1035] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:43.093767 29073 tablet_bootstrap.cc:492] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.094250 28708 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.094417 29075 tablet_bootstrap.cc:492] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:43.094738 29073 tablet_bootstrap.cc:654] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.095448 29075 tablet_bootstrap.cc:654] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.095636 29073 tablet_bootstrap.cc:492] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:43.095714 29073 ts_tablet_manager.cc:1367] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:43.096127 29073 raft_consensus.cc:348] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.096192 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 938581084b27405cbc01917e1a982208 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:43.096225 29073 raft_consensus.cc:374] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.096254 29073 raft_consensus.cc:729] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.096386 29073 consensus_queue.cc:260] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.096650 29073 ts_tablet_manager.cc:1398] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.096578 25032 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.099047 24485 catalog_manager.cc:5484] T 938581084b27405cbc01917e1a982208 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.100335 27312 raft_consensus.cc:2932] T 938581084b27405cbc01917e1a982208 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:43.101867 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 938581084b27405cbc01917e1a982208 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:43.102103 27864 tablet_replica.cc:330] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:43.102232 27864 raft_consensus.cc:2229] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.102365 27864 pending_rounds.cc:62] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Trying to abort 1 pending ops.
I20240502 01:58:43.102443 27864 pending_rounds.cc:68] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023063544053649408 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "938581084b27405cbc01917e1a982208" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } } }
I20240502 01:58:43.102587 27864 raft_consensus.cc:2894] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:58:43.102658 27864 raft_consensus.cc:2866] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:58:43.102706 27864 raft_consensus.cc:2258] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.102896 29075 tablet_bootstrap.cc:492] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:43.102983 29075 ts_tablet_manager.cc:1367] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
W20240502 01:58:43.103503 29071 consensus_queue.cc:1159] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "92cb785c165749eca49af5784756461c" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 01:58:43.103505 29075 raft_consensus.cc:348] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.103601 29075 raft_consensus.cc:374] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.104218 29075 raft_consensus.cc:729] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.104413 29075 consensus_queue.cc:260] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.104533 27864 ts_tablet_manager.cc:1875] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:43.105413 29075 ts_tablet_manager.cc:1398] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:43.106565 27864 ts_tablet_manager.cc:1888] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:43.106634 27864 log.cc:1196] T 938581084b27405cbc01917e1a982208 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/938581084b27405cbc01917e1a982208
I20240502 01:58:43.107153 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 938581084b27405cbc01917e1a982208 (table rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]) successfully deleted
I20240502 01:58:43.107298 29074 tablet_bootstrap.cc:492] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:43.108330 28941 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.108501 29074 tablet_bootstrap.cc:654] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.109287 28941 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:43.110100 29074 tablet_bootstrap.cc:492] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:43.110183 29074 ts_tablet_manager.cc:1367] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:43.110589 29074 raft_consensus.cc:348] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.110679 29074 raft_consensus.cc:374] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.110699 29074 raft_consensus.cc:729] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.110800 29074 consensus_queue.cc:260] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.111016 29074 ts_tablet_manager.cc:1398] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.113009 28941 ts_tablet_manager.cc:1367] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.050s user 0.024s sys 0.000s
I20240502 01:58:43.113415 28941 raft_consensus.cc:348] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:43.113523 28941 raft_consensus.cc:729] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:43.113600 28941 consensus_queue.cc:260] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:43.113989 28941 ts_tablet_manager.cc:1398] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.118350 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-63312ec15b034e47bc2dd484b57d4648 received from {username='slave'} at 127.17.243.5:47383
I20240502 01:58:43.118425 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-63312ec15b034e47bc2dd484b57d4648 on tablet 63312ec15b034e47bc2dd484b57d4648 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:43.123955 29079 ts_tablet_manager.cc:902] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: Initiating tablet copy from peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:43.124635 29079 tablet_copy_client.cc:308] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.8:44673
I20240502 01:58:43.125291 27599 tablet_copy_service.cc:140] P bb653087e8554c7591cf547dd969f3be: Received BeginTabletCopySession request for tablet 823fe241dc1b40cca121a22d669e4226 from peer e0b9e7df4e8543fa9b8e3ffc24330657 ({username='slave'} at 127.17.243.4:47413)
I20240502 01:58:43.125445 27599 tablet_copy_service.cc:161] P bb653087e8554c7591cf547dd969f3be: Beginning new tablet copy session on tablet 823fe241dc1b40cca121a22d669e4226 from peer e0b9e7df4e8543fa9b8e3ffc24330657 at {username='slave'} at 127.17.243.4:47413: session id = e0b9e7df4e8543fa9b8e3ffc24330657-823fe241dc1b40cca121a22d669e4226
I20240502 01:58:43.126802 27599 tablet_copy_source_session.cc:205] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.128238 29079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823fe241dc1b40cca121a22d669e4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.130219 29079 tablet_copy_client.cc:783] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.130395 29079 tablet_copy_client.cc:647] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.131999 29079 tablet_copy_client.cc:523] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.133436 29079 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:43.133734 28930 ts_tablet_manager.cc:902] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:43.134142 28930 tablet_copy_client.cc:308] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:43.134562 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet ebe8db639839433d967583371351eacb from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:39965)
I20240502 01:58:43.134642 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet ebe8db639839433d967583371351eacb from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:39965: session id = bb653087e8554c7591cf547dd969f3be-ebe8db639839433d967583371351eacb
I20240502 01:58:43.135447 24785 tablet_copy_source_session.cc:205] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.136340 28930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8db639839433d967583371351eacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.138124 28872 consensus_queue.cc:1035] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.138234 29079 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.138274 28930 tablet_copy_client.cc:783] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.138422 28930 tablet_copy_client.cc:647] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.138468 29079 tablet_bootstrap.cc:492] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap complete.
I20240502 01:58:43.138653 29079 ts_tablet_manager.cc:1367] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.006s sys 0.000s
I20240502 01:58:43.139091 29079 raft_consensus.cc:348] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:43.139263 29079 raft_consensus.cc:729] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: LEARNER
I20240502 01:58:43.139411 29079 consensus_queue.cc:260] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:43.139819 28930 tablet_copy_client.cc:523] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.140180 27578 raft_consensus.cc:3037] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.141044 28930 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:43.141269 24630 consensus_queue.cc:237] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.142185 28447 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.142799 24765 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.142824 24897 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.142892 29079 ts_tablet_manager.cc:1398] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:43.143838 27599 tablet_copy_service.cc:342] P bb653087e8554c7591cf547dd969f3be: Request end of tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-823fe241dc1b40cca121a22d669e4226 received from {username='slave'} at 127.17.243.4:47413
I20240502 01:58:43.143920 27599 tablet_copy_service.cc:433] P bb653087e8554c7591cf547dd969f3be: ending tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-823fe241dc1b40cca121a22d669e4226 on tablet 823fe241dc1b40cca121a22d669e4226 with peer e0b9e7df4e8543fa9b8e3ffc24330657
W20240502 01:58:43.145522 24567 consensus_peers.cc:485] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 645305c993a5486ebc1e6c8b81941128. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.145875 28489 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.146262 28489 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.148733 28447 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.148937 27865 consensus_queue.cc:1035] T 584cbed9fc494fc2b32c1bfd2571d88b P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.151207 27747 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.153738 24486 catalog_manager.cc:5484] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.153736 24765 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.153944 24897 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.165205 24897 consensus_queue.cc:237] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.166571 25032 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.166750 24765 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:43.166817 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d0cc9aba3804968a6dce97682f6b38e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.167008 28779 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.167340 28779 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.168736 27093 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.168995 28930 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.169199 28930 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:43.169194 25032 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.169320 28930 ts_tablet_manager.cc:1367] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.004s
I20240502 01:58:43.169591 28930 raft_consensus.cc:348] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.169682 28930 raft_consensus.cc:729] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:43.169757 28930 consensus_queue.cc:260] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.170006 28930 ts_tablet_manager.cc:1398] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.170490 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-ebe8db639839433d967583371351eacb received from {username='slave'} at 127.17.243.8:39965
I20240502 01:58:43.170554 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session bb653087e8554c7591cf547dd969f3be-ebe8db639839433d967583371351eacb on tablet ebe8db639839433d967583371351eacb with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:43.172209 28903 ts_tablet_manager.cc:902] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:43.172627 28903 tablet_copy_client.cc:308] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:43.172616 24483 catalog_manager.cc:5484] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.174571 24765 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.181200 25032 consensus_queue.cc:237] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.182516 24630 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.182776 24897 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.183020 27894 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:43.183164 24966 consensus_peers.cc:485] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b8667a49f1a44339122040b0122be0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.183634 27894 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.185161 27890 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.185572 24897 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.185847 24630 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.188393 24486 catalog_manager.cc:5484] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.194954 24630 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.196154 24765 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:43.196156 27177 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:43.196514 28799 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:58:43.196923 24567 consensus_peers.cc:485] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e720d8363bb049938798ab0b9aabc28d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.197659 28799 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:43.199101 28799 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.200055 27177 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.200964 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 6560bc797c024a90aa87515d0ef2c4c6 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:38617)
I20240502 01:58:43.201040 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 6560bc797c024a90aa87515d0ef2c4c6 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:38617: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-6560bc797c024a90aa87515d0ef2c4c6
I20240502 01:58:43.201447 24765 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.201782 25054 tablet_copy_source_session.cc:205] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.202708 28903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6560bc797c024a90aa87515d0ef2c4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.202890 24483 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.204782 28903 tablet_copy_client.cc:783] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.204908 28903 tablet_copy_client.cc:647] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.206104 28903 tablet_copy_client.cc:523] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.207121 28903 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:43.209895 28903 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.210124 28903 tablet_bootstrap.cc:492] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:43.210263 28903 ts_tablet_manager.cc:1367] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:43.210667 28903 raft_consensus.cc:348] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.210776 28903 raft_consensus.cc:729] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:43.210891 28903 consensus_queue.cc:260] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.211162 28903 ts_tablet_manager.cc:1398] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.212420 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-6560bc797c024a90aa87515d0ef2c4c6 received from {username='slave'} at 127.17.243.7:38617
I20240502 01:58:43.212486 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-6560bc797c024a90aa87515d0ef2c4c6 on tablet 6560bc797c024a90aa87515d0ef2c4c6 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:43.215690 24765 consensus_queue.cc:237] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:43.216900 25032 raft_consensus.cc:1261] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.216888 24630 raft_consensus.cc:1261] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.217252 28393 consensus_queue.cc:1035] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.217432 28404 consensus_queue.cc:1035] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:43.217603 24700 consensus_peers.cc:485] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2284e66dfe44567b2170e0fcff27bd6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.219146 28404 raft_consensus.cc:2932] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:43.219561 24630 raft_consensus.cc:2932] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:43.219708 25032 raft_consensus.cc:2932] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:43.222028 24479 catalog_manager.cc:5484] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:43.224155 27578 raft_consensus.cc:1203] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.224892 28447 raft_consensus.cc:482] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:43.224982 28447 raft_consensus.cc:504] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.225332 28447 leader_election.cc:288] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:43.225630 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51462d8e7b414a169f3cfedb6b7ca585" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:43.225692 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51462d8e7b414a169f3cfedb6b7ca585" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:43.225757 27312 raft_consensus.cc:2454] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:43.225804 27177 raft_consensus.cc:2454] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:43.226080 27377 leader_election.cc:302] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:43.226423 28447 raft_consensus.cc:2781] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:43.226478 28447 raft_consensus.cc:482] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:43.226512 28447 raft_consensus.cc:3037] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.228375 28447 raft_consensus.cc:504] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.228735 28447 leader_election.cc:288] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:43.229004 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51462d8e7b414a169f3cfedb6b7ca585" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:43.229058 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51462d8e7b414a169f3cfedb6b7ca585" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:43.229115 27312 raft_consensus.cc:3037] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.229142 27177 raft_consensus.cc:3037] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.230003 27177 raft_consensus.cc:2454] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:43.230015 27312 raft_consensus.cc:2454] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:43.230270 27379 leader_election.cc:302] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:43.230530 28447 raft_consensus.cc:2781] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:43.230615 28447 raft_consensus.cc:686] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:43.230728 28447 consensus_queue.cc:237] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.230988 24897 consensus_queue.cc:237] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.232306 24630 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.232811 27645 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.233487 24486 catalog_manager.cc:5484] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.233722 24765 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:43.234288 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e47667fc8fa49e6b18ee3df06cda00a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.234592 28708 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.234870 27093 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.235303 24630 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.238184 24486 catalog_manager.cc:5484] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.239334 24765 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.245450 24897 consensus_queue.cc:237] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.246688 24765 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.246762 25032 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.247027 27093 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:43.247216 24833 consensus_peers.cc:485] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d5f6bcbe0a048eb85b9475348b155ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.247225 28779 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.248754 28779 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.249230 25032 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.249487 24765 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:43.251591 24486 catalog_manager.cc:5484] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.257745 25032 consensus_queue.cc:237] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.258872 24897 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.258950 24765 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.259267 28534 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:43.259418 24966 consensus_peers.cc:485] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 086c9b674a0543cab3c82349d151e373. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.259445 27890 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.260953 27894 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.261402 24897 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.261384 24765 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.263556 24483 catalog_manager.cc:5484] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:43.270550 24897 consensus_queue.cc:237] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.271829 29174 ts_tablet_manager.cc:902] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:43.271883 24630 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:43.272063 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16ae91433f10465c922183298a387caa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.272230 24765 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.272266 27093 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.272362 29174 tablet_copy_client.cc:308] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:43.272616 27645 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.272764 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 693e0959e61c44c59f797e4060187cef from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:43875)
I20240502 01:58:43.272841 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 693e0959e61c44c59f797e4060187cef from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:43875: session id = 1049b4c8783b48f39be22bd113ad349f-693e0959e61c44c59f797e4060187cef
I20240502 01:58:43.273471 25054 tablet_copy_source_session.cc:205] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.274077 27645 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.274261 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693e0959e61c44c59f797e4060187cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.275208 24630 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.276661 29174 tablet_copy_client.cc:783] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.276993 24765 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.283035 24479 catalog_manager.cc:5484] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.284698 29174 tablet_copy_client.cc:647] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.286386 29174 tablet_copy_client.cc:523] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.288111 29174 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.312685 29174 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.312935 29174 tablet_bootstrap.cc:492] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:43.313108 29174 ts_tablet_manager.cc:1367] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240502 01:58:43.313956 27446 raft_consensus.cc:1203] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.314673 29174 raft_consensus.cc:348] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.315034 29174 raft_consensus.cc:729] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:43.315192 29174 consensus_queue.cc:260] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.315671 29174 ts_tablet_manager.cc:1398] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.316572 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-693e0959e61c44c59f797e4060187cef received from {username='slave'} at 127.17.243.6:43875
I20240502 01:58:43.316653 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-693e0959e61c44c59f797e4060187cef on tablet 693e0959e61c44c59f797e4060187cef with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:43.350670 28315 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.353426 28447 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.353629 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:43.358695 27312 raft_consensus.cc:1261] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:43.358752 27177 raft_consensus.cc:1261] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:43.359195 28447 consensus_queue.cc:1035] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.359534 28315 consensus_queue.cc:1035] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:43.361860 24486 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 51462d8e7b414a169f3cfedb6b7ca585 (table rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d]). Expected version 1 got 0
I20240502 01:58:43.362545 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 51462d8e7b414a169f3cfedb6b7ca585 (table rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d]). Expected version 1 got 0
I20240502 01:58:43.363037 29247 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:43.363443 29247 tablet.cc:1620] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:43.363834 24483 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 51462d8e7b414a169f3cfedb6b7ca585 (table rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d]). Expected version 1 got 0
I20240502 01:58:43.364269 28851 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:43.364524 28851 tablet.cc:1620] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:43.365943 28842 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:43.366222 28842 tablet.cc:1620] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:43.367295 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d] alter complete (version 1)
I20240502 01:58:43.402825 24897 consensus_queue.cc:237] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.404033 24765 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:43.404127 25032 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:43.404249 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48085cebf94e4f549e6174a88acdcc59. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.404451 28396 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.404619 27093 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:43.406164 28779 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.406580 25032 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.407646 24765 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:43.409674 24486 catalog_manager.cc:5484] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.432942 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:43.438803 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 73b38dc3e25640c0a6a8dd389ec67c1b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.439062 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73b38dc3e25640c0a6a8dd389ec67c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.439950 24611 tablet_service.cc:1467] Processing CreateTablet for tablet abfa462d7c6d43f0ac967d298ee9050e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.440202 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa462d7c6d43f0ac967d298ee9050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.440171 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 73b38dc3e25640c0a6a8dd389ec67c1b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.440423 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73b38dc3e25640c0a6a8dd389ec67c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.440420 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 73b38dc3e25640c0a6a8dd389ec67c1b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.440670 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73b38dc3e25640c0a6a8dd389ec67c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.441551 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 84ae33b0741c48c090dda6d4a819ef02 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.441839 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae33b0741c48c090dda6d4a819ef02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.442189 27558 tablet_service.cc:1467] Processing CreateTablet for tablet abfa462d7c6d43f0ac967d298ee9050e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.442389 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa462d7c6d43f0ac967d298ee9050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.442864 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 84ae33b0741c48c090dda6d4a819ef02 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.443084 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae33b0741c48c090dda6d4a819ef02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.443884 29075 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:43.444103 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 84ae33b0741c48c090dda6d4a819ef02 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.444366 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae33b0741c48c090dda6d4a819ef02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.444795 29075 tablet_bootstrap.cc:654] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.445873 29314 tablet_bootstrap.cc:492] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:43.446307 29073 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.446947 29314 tablet_bootstrap.cc:654] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.447131 29073 tablet_bootstrap.cc:654] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.447840 27155 tablet_service.cc:1467] Processing CreateTablet for tablet abfa462d7c6d43f0ac967d298ee9050e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:43.448043 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa462d7c6d43f0ac967d298ee9050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.448269 29073 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:43.448355 29073 ts_tablet_manager.cc:1367] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:43.448437 29314 tablet_bootstrap.cc:492] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:43.448550 29314 ts_tablet_manager.cc:1367] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:43.448827 29073 raft_consensus.cc:348] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.448928 29073 raft_consensus.cc:374] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.448959 29073 raft_consensus.cc:729] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.448997 29314 raft_consensus.cc:348] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.449074 29314 raft_consensus.cc:374] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.449103 29314 raft_consensus.cc:729] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.449071 29073 consensus_queue.cc:260] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.449211 29314 consensus_queue.cc:260] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.449326 29073 ts_tablet_manager.cc:1398] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.449548 29314 ts_tablet_manager.cc:1398] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.449764 29314 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:43.450323 29313 tablet_bootstrap.cc:492] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:43.450847 29314 tablet_bootstrap.cc:654] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.450871 29312 tablet_bootstrap.cc:492] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:43.451701 29312 tablet_bootstrap.cc:654] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.452628 29314 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:43.452702 29314 ts_tablet_manager.cc:1367] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:43.453073 29075 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:43.453150 29075 ts_tablet_manager.cc:1367] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:43.453104 29314 raft_consensus.cc:348] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.453183 29314 raft_consensus.cc:374] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.453210 29314 raft_consensus.cc:729] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.453320 29314 consensus_queue.cc:260] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.453565 29075 raft_consensus.cc:348] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.453655 29075 raft_consensus.cc:374] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.453683 29075 raft_consensus.cc:729] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.453809 29075 consensus_queue.cc:260] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.454092 29075 ts_tablet_manager.cc:1398] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:43.454157 29314 ts_tablet_manager.cc:1398] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.454246 29075 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:43.454289 29314 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:43.454738 29313 tablet_bootstrap.cc:654] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.455052 29314 tablet_bootstrap.cc:654] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.455049 29075 tablet_bootstrap.cc:654] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.455646 29313 tablet_bootstrap.cc:492] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:43.455714 29313 ts_tablet_manager.cc:1367] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:43.456073 29313 raft_consensus.cc:348] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.456166 29313 raft_consensus.cc:374] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.456192 29313 raft_consensus.cc:729] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.456283 29313 consensus_queue.cc:260] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.456522 29313 ts_tablet_manager.cc:1398] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.456779 29314 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:43.456849 29314 ts_tablet_manager.cc:1367] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:43.457239 29314 raft_consensus.cc:348] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.457293 29075 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:43.457334 29314 raft_consensus.cc:374] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.457361 29314 raft_consensus.cc:729] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.457379 29075 ts_tablet_manager.cc:1367] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:43.457474 29314 consensus_queue.cc:260] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.457743 29314 ts_tablet_manager.cc:1398] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.457762 29075 raft_consensus.cc:348] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.457844 29075 raft_consensus.cc:374] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.457873 29075 raft_consensus.cc:729] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.458006 29075 consensus_queue.cc:260] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.462302 29075 ts_tablet_manager.cc:1398] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:43.466616 29312 tablet_bootstrap.cc:492] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:43.466711 29312 ts_tablet_manager.cc:1367] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240502 01:58:43.467144 29312 raft_consensus.cc:348] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.467242 29312 raft_consensus.cc:374] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.467276 29312 raft_consensus.cc:729] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.467391 29312 consensus_queue.cc:260] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.467716 29312 ts_tablet_manager.cc:1398] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.467880 29312 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:43.468711 29312 tablet_bootstrap.cc:654] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.471077 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:43.476953 29247 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:43.477244 29247 tablet.cc:1620] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:43.477334 28851 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:43.477595 28851 tablet.cc:1620] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:43.477566 28842 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:43.477834 28842 tablet.cc:1620] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:43.478466 29312 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:43.478543 29312 ts_tablet_manager.cc:1367] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:58:43.478928 29312 raft_consensus.cc:348] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.479036 29312 raft_consensus.cc:374] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.479066 29312 raft_consensus.cc:729] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.479184 29312 consensus_queue.cc:260] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.479437 29312 ts_tablet_manager.cc:1398] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:43.482805 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=48af563d7bbb4a04b81926ea6791a88d] alter complete (version 2)
I20240502 01:58:43.487495 27578 raft_consensus.cc:1203] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.505402 29102 raft_consensus.cc:482] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:43.505511 29102 raft_consensus.cc:504] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.505895 29102 leader_election.cc:288] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:43.506150 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:43.506260 27578 raft_consensus.cc:2454] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:43.506528 27246 leader_election.cc:302] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:43.506701 29102 raft_consensus.cc:2781] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:43.506762 29102 raft_consensus.cc:482] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:43.506803 29102 raft_consensus.cc:3037] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.507032 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:43.507169 27177 raft_consensus.cc:2454] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:43.507702 29102 raft_consensus.cc:504] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.508015 29102 leader_election.cc:288] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:43.508296 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:43.508296 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:43.508402 27177 raft_consensus.cc:3037] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.508402 27578 raft_consensus.cc:3037] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.509251 27177 raft_consensus.cc:2454] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:43.509279 27578 raft_consensus.cc:2454] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:43.509512 27246 leader_election.cc:302] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:43.509660 29102 raft_consensus.cc:2781] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:43.509730 29102 raft_consensus.cc:686] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:43.509848 29102 consensus_queue.cc:237] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:43.511787 24479 catalog_manager.cc:5484] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.523118 25032 raft_consensus.cc:1203] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:43.550035 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 01:58:43.550139 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 01:58:43.552624 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet e2284e66dfe44567b2170e0fcff27bd6
I20240502 01:58:43.552975 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 61a7e0e4e54f47849bb29d1e7ae029af
I20240502 01:58:43.553253 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet ee784fe0525240078fc79d0af2756b8d
I20240502 01:58:43.553548 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:43.553725 28027 tablet_replica.cc:330] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:43.553835 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:43.553881 28027 raft_consensus.cc:2229] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.554073 28027 raft_consensus.cc:2258] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.554221 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:43.554463 29390 tablet_replica.cc:330] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:43.554549 28027 ts_tablet_manager.cc:1875] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.554589 27864 tablet_replica.cc:330] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:43.554577 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:43.554598 29390 raft_consensus.cc:2229] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.554710 29390 raft_consensus.cc:2258] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.554706 27864 raft_consensus.cc:2229] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.554941 27864 raft_consensus.cc:2258] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.555153 29390 ts_tablet_manager.cc:1875] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.555271 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:43.555402 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a7e0e4e54f47849bb29d1e7ae029af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:43.555431 25010 tablet_service.cc:1514] Processing DeleteTablet for tablet ee784fe0525240078fc79d0af2756b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:43.555518 27740 tablet_replica.cc:330] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:43.555639 27740 raft_consensus.cc:2229] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 01:58:43.555722 24473 catalog_manager.cc:4797] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): delete failed for tablet e2284e66dfe44567b2170e0fcff27bd6 because the tablet was not found. No further retry: Not found: Tablet not found: e2284e66dfe44567b2170e0fcff27bd6
I20240502 01:58:43.555773 27740 raft_consensus.cc:2258] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.556190 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet ee784fe0525240078fc79d0af2756b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:43.556506 27740 ts_tablet_manager.cc:1875] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.556588 28027 ts_tablet_manager.cc:1888] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:43.556643 28027 log.cc:1196] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/61a7e0e4e54f47849bb29d1e7ae029af
I20240502 01:58:43.556780 27864 ts_tablet_manager.cc:1875] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.556880 28027 ts_tablet_manager.cc:1909] T 61a7e0e4e54f47849bb29d1e7ae029af P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:43.557247 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet e2284e66dfe44567b2170e0fcff27bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:43.557399 28540 tablet_replica.cc:330] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:43.557518 28540 raft_consensus.cc:2229] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.557672 28540 raft_consensus.cc:2258] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.557868 29390 ts_tablet_manager.cc:1888] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:43.557950 29390 log.cc:1196] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/61a7e0e4e54f47849bb29d1e7ae029af
I20240502 01:58:43.558187 29390 ts_tablet_manager.cc:1909] T 61a7e0e4e54f47849bb29d1e7ae029af P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:43.558512 28540 ts_tablet_manager.cc:1875] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.558620 27740 ts_tablet_manager.cc:1888] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.558673 27740 log.cc:1196] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/e2284e66dfe44567b2170e0fcff27bd6
I20240502 01:58:43.559041 27864 ts_tablet_manager.cc:1888] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.559101 27864 log.cc:1196] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/e2284e66dfe44567b2170e0fcff27bd6
I20240502 01:58:43.559144 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet ee784fe0525240078fc79d0af2756b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:43.559302 27864 ts_tablet_manager.cc:1909] T e2284e66dfe44567b2170e0fcff27bd6 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:43.559475 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 61a7e0e4e54f47849bb29d1e7ae029af (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.559604 29174 ts_tablet_manager.cc:902] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:43.558916 27740 ts_tablet_manager.cc:1909] T e2284e66dfe44567b2170e0fcff27bd6 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:43.560017 29174 tablet_copy_client.cc:308] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:43.560292 27740 tablet_replica.cc:330] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:43.560443 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 7d0cc9aba3804968a6dce97682f6b38e from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:44697)
I20240502 01:58:43.560510 27740 raft_consensus.cc:2229] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.560528 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 7d0cc9aba3804968a6dce97682f6b38e from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:44697: session id = 1049b4c8783b48f39be22bd113ad349f-7d0cc9aba3804968a6dce97682f6b38e
I20240502 01:58:43.560586 28540 ts_tablet_manager.cc:1888] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.560657 28540 log.cc:1196] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/e2284e66dfe44567b2170e0fcff27bd6
I20240502 01:58:43.560767 27740 raft_consensus.cc:2258] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.560897 28540 ts_tablet_manager.cc:1909] T e2284e66dfe44567b2170e0fcff27bd6 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:43.561215 27740 ts_tablet_manager.cc:1875] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.561319 24918 tablet_copy_source_session.cc:205] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.561482 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet e2284e66dfe44567b2170e0fcff27bd6 (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.561573 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet e2284e66dfe44567b2170e0fcff27bd6 (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.561901 28540 tablet_replica.cc:330] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:43.562026 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 61a7e0e4e54f47849bb29d1e7ae029af (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.562073 28540 raft_consensus.cc:2229] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.562208 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0cc9aba3804968a6dce97682f6b38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.562258 28540 raft_consensus.cc:2258] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.562661 28540 ts_tablet_manager.cc:1875] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.563766 29174 tablet_copy_client.cc:783] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.563915 29174 tablet_copy_client.cc:647] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.564148 27740 ts_tablet_manager.cc:1888] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:43.564209 27740 log.cc:1196] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/61a7e0e4e54f47849bb29d1e7ae029af
I20240502 01:58:43.564330 28540 ts_tablet_manager.cc:1888] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:43.564384 28540 log.cc:1196] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee784fe0525240078fc79d0af2756b8d
I20240502 01:58:43.564431 27740 ts_tablet_manager.cc:1909] T 61a7e0e4e54f47849bb29d1e7ae029af P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:43.564606 28540 ts_tablet_manager.cc:1909] T ee784fe0525240078fc79d0af2756b8d P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:43.564994 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 61a7e0e4e54f47849bb29d1e7ae029af (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.565100 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet ee784fe0525240078fc79d0af2756b8d (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.565203 27740 tablet_replica.cc:330] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:43.565317 27740 raft_consensus.cc:2229] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.565459 27740 raft_consensus.cc:2258] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.565616 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet e2284e66dfe44567b2170e0fcff27bd6 (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.565735 29174 tablet_copy_client.cc:523] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.565954 27740 ts_tablet_manager.cc:1875] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.566781 29174 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.567435 27740 ts_tablet_manager.cc:1888] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:43.567490 27740 log.cc:1196] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee784fe0525240078fc79d0af2756b8d
I20240502 01:58:43.567675 27740 ts_tablet_manager.cc:1909] T ee784fe0525240078fc79d0af2756b8d P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:43.568192 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet ee784fe0525240078fc79d0af2756b8d (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.569223 27864 tablet_replica.cc:330] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:43.569340 27864 raft_consensus.cc:2229] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.569448 27864 raft_consensus.cc:2258] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.569893 27864 ts_tablet_manager.cc:1875] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.571652 28903 ts_tablet_manager.cc:902] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:43.572046 28903 tablet_copy_client.cc:308] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:43.572449 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 9b8667a49f1a44339122040b0122be0d from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:38617)
I20240502 01:58:43.572544 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 9b8667a49f1a44339122040b0122be0d from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:38617: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-9b8667a49f1a44339122040b0122be0d
I20240502 01:58:43.573300 25054 tablet_copy_source_session.cc:205] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.574136 28903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8667a49f1a44339122040b0122be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.576011 28903 tablet_copy_client.cc:783] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.576153 28903 tablet_copy_client.cc:647] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.578111 28903 tablet_copy_client.cc:523] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.579203 28903 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:43.582232 27864 ts_tablet_manager.cc:1888] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:43.582309 27864 log.cc:1196] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee784fe0525240078fc79d0af2756b8d
I20240502 01:58:43.582765 27864 ts_tablet_manager.cc:1909] T ee784fe0525240078fc79d0af2756b8d P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:43.583456 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet ee784fe0525240078fc79d0af2756b8d (table rebalance_test_table_14 [id=546c4ab085564c55a070f5e4d151b8eb]) successfully deleted
I20240502 01:58:43.585755 29174 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.586045 29174 tablet_bootstrap.cc:492] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:43.586191 29174 ts_tablet_manager.cc:1367] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.004s
I20240502 01:58:43.586524 29174 raft_consensus.cc:348] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.586616 29174 raft_consensus.cc:729] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:43.586727 29174 consensus_queue.cc:260] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.587239 29174 ts_tablet_manager.cc:1398] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.587926 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-7d0cc9aba3804968a6dce97682f6b38e received from {username='slave'} at 127.17.243.6:44697
I20240502 01:58:43.587991 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-7d0cc9aba3804968a6dce97682f6b38e on tablet 7d0cc9aba3804968a6dce97682f6b38e with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:43.588481 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:43.592984 29247 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:43.593686 28851 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:43.594738 28842 tablet.cc:1722] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:43.598040 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=48af563d7bbb4a04b81926ea6791a88d] alter complete (version 3)
I20240502 01:58:43.603286 28874 raft_consensus.cc:482] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:43.603368 28874 raft_consensus.cc:504] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.603705 28874 leader_election.cc:288] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:43.603965 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa462d7c6d43f0ac967d298ee9050e" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:43.604080 24630 raft_consensus.cc:2454] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:43.604090 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa462d7c6d43f0ac967d298ee9050e" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:43.604182 27177 raft_consensus.cc:2454] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:43.604367 27513 leader_election.cc:302] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:43.604610 28874 raft_consensus.cc:2781] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:43.604658 28874 raft_consensus.cc:482] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:43.604693 28874 raft_consensus.cc:3037] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.605368 28874 raft_consensus.cc:504] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.605670 28874 leader_election.cc:288] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:43.606020 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa462d7c6d43f0ac967d298ee9050e" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:43.606112 27177 raft_consensus.cc:3037] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.606791 27177 raft_consensus.cc:2454] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:43.606995 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfa462d7c6d43f0ac967d298ee9050e" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:43.607069 27513 leader_election.cc:302] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:43.607097 24630 raft_consensus.cc:3037] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.607825 24630 raft_consensus.cc:2454] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:43.608258 28874 raft_consensus.cc:2781] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:43.607729 29403 ts_tablet_manager.cc:902] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:43.608337 28874 raft_consensus.cc:686] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:43.608448 28874 consensus_queue.cc:237] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:43.610116 24479 catalog_manager.cc:5484] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:43.612447 29403 tablet_copy_client.cc:308] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:43.612833 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 645305c993a5486ebc1e6c8b81941128 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:51949)
I20240502 01:58:43.612918 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 645305c993a5486ebc1e6c8b81941128 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:51949: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-645305c993a5486ebc1e6c8b81941128
I20240502 01:58:43.613590 24652 tablet_copy_source_session.cc:205] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.614287 29403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645305c993a5486ebc1e6c8b81941128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.614567 28903 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.614881 28903 tablet_bootstrap.cc:492] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:43.615062 28903 ts_tablet_manager.cc:1367] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.000s
I20240502 01:58:43.615463 28903 raft_consensus.cc:348] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.615613 28903 raft_consensus.cc:729] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:43.615734 28903 consensus_queue.cc:260] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.615913 29403 tablet_copy_client.cc:783] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.616153 29403 tablet_copy_client.cc:647] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.616494 28903 ts_tablet_manager.cc:1398] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.617075 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-9b8667a49f1a44339122040b0122be0d received from {username='slave'} at 127.17.243.7:38617
I20240502 01:58:43.617137 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-9b8667a49f1a44339122040b0122be0d on tablet 9b8667a49f1a44339122040b0122be0d with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:43.617120 29403 tablet_copy_client.cc:523] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.618069 29403 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:43.621645 27177 raft_consensus.cc:1203] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.622958 29174 ts_tablet_manager.cc:902] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:43.623311 29174 tablet_copy_client.cc:308] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:43.623704 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 0e47667fc8fa49e6b18ee3df06cda00a from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:44697)
I20240502 01:58:43.623786 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 0e47667fc8fa49e6b18ee3df06cda00a from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:44697: session id = 1049b4c8783b48f39be22bd113ad349f-0e47667fc8fa49e6b18ee3df06cda00a
I20240502 01:58:43.624430 24918 tablet_copy_source_session.cc:205] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.625216 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e47667fc8fa49e6b18ee3df06cda00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.626718 29174 tablet_copy_client.cc:783] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.626843 29174 tablet_copy_client.cc:647] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.627816 29174 tablet_copy_client.cc:523] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.628646 29174 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.637523 29403 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.637733 29403 tablet_bootstrap.cc:492] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:43.637863 29403 ts_tablet_manager.cc:1367] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20240502 01:58:43.638234 29403 raft_consensus.cc:348] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.638325 29403 raft_consensus.cc:729] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:43.638415 29403 consensus_queue.cc:260] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.638682 29403 ts_tablet_manager.cc:1398] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:58:43.639160 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-645305c993a5486ebc1e6c8b81941128 received from {username='slave'} at 127.17.243.7:51949
I20240502 01:58:43.639226 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-645305c993a5486ebc1e6c8b81941128 on tablet 645305c993a5486ebc1e6c8b81941128 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:43.640255 27017 raft_consensus.cc:1050] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:43.640625 27017 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:43.642030 27578 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.642113 24765 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.642382 24896 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.643016 28534 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.643163 27894 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.643441 27894 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.645793 27890 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.646656 24897 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.648241 24479 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:43.648553 24765 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.648772 29174 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.649006 29174 tablet_bootstrap.cc:492] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:43.649171 29174 ts_tablet_manager.cc:1367] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.021s user 0.021s sys 0.000s
I20240502 01:58:43.649803 29174 raft_consensus.cc:348] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.649962 29174 raft_consensus.cc:729] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:43.650092 29174 consensus_queue.cc:260] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.650437 29174 ts_tablet_manager.cc:1398] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.650900 27578 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.652318 25032 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:43.653239 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-0e47667fc8fa49e6b18ee3df06cda00a received from {username='slave'} at 127.17.243.6:44697
I20240502 01:58:43.653309 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-0e47667fc8fa49e6b18ee3df06cda00a on tablet 0e47667fc8fa49e6b18ee3df06cda00a with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:43.653640 27578 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:43.653774 24765 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:43.654610 28534 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:43.654734 27894 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:43.656111 28534 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.656572 24765 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.656873 27578 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.657498 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 488816b0d5d542bcb5fa12ff7e70fd40 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:43.659082 24485 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:43.661160 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 488816b0d5d542bcb5fa12ff7e70fd40 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:43.661339 28027 tablet_replica.cc:330] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:43.661458 28027 raft_consensus.cc:2229] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.661617 28027 raft_consensus.cc:2258] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.662228 28027 ts_tablet_manager.cc:1875] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:43.663681 28027 ts_tablet_manager.cc:1888] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:43.663741 28027 log.cc:1196] T 488816b0d5d542bcb5fa12ff7e70fd40 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/488816b0d5d542bcb5fa12ff7e70fd40
I20240502 01:58:43.664130 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 488816b0d5d542bcb5fa12ff7e70fd40 (table rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]) successfully deleted
I20240502 01:58:43.672880 29174 ts_tablet_manager.cc:902] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:43.673200 29174 tablet_copy_client.cc:308] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:43.673657 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 16ae91433f10465c922183298a387caa from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:44697)
I20240502 01:58:43.673748 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 16ae91433f10465c922183298a387caa from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:44697: session id = 1049b4c8783b48f39be22bd113ad349f-16ae91433f10465c922183298a387caa
I20240502 01:58:43.674542 24918 tablet_copy_source_session.cc:205] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.675037 27446 raft_consensus.cc:1203] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:43.675405 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ae91433f10465c922183298a387caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.676946 29174 tablet_copy_client.cc:783] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.677062 29174 tablet_copy_client.cc:647] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.678135 29174 tablet_copy_client.cc:523] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.678920 29174 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.685460 29424 ts_tablet_manager.cc:902] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:43.686025 29424 tablet_copy_client.cc:308] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:43.686452 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet e720d8363bb049938798ab0b9aabc28d from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:44271)
I20240502 01:58:43.686547 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet e720d8363bb049938798ab0b9aabc28d from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:44271: session id = bb653087e8554c7591cf547dd969f3be-e720d8363bb049938798ab0b9aabc28d
I20240502 01:58:43.687352 24652 tablet_copy_source_session.cc:205] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.688113 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e720d8363bb049938798ab0b9aabc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.689790 29424 tablet_copy_client.cc:783] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.690037 29424 tablet_copy_client.cc:647] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.691097 29424 tablet_copy_client.cc:523] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.692282 29424 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:43.694499 29403 ts_tablet_manager.cc:902] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:43.694922 29403 tablet_copy_client.cc:308] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:43.695530 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 8d5f6bcbe0a048eb85b9475348b155ef from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:55633)
I20240502 01:58:43.695627 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 8d5f6bcbe0a048eb85b9475348b155ef from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:55633: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-8d5f6bcbe0a048eb85b9475348b155ef
I20240502 01:58:43.696447 24918 tablet_copy_source_session.cc:205] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.697193 29403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5f6bcbe0a048eb85b9475348b155ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.698985 29403 tablet_copy_client.cc:783] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.699117 29403 tablet_copy_client.cc:647] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.699792 29174 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.699817 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:43.699925 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:43.700013 29174 tablet_bootstrap.cc:492] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:43.700147 29174 ts_tablet_manager.cc:1367] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20240502 01:58:43.700533 29174 raft_consensus.cc:348] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.700639 29174 raft_consensus.cc:729] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:43.700732 29174 consensus_queue.cc:260] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:43.701671 29174 ts_tablet_manager.cc:1398] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:58:43.702246 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-16ae91433f10465c922183298a387caa received from {username='slave'} at 127.17.243.6:44697
I20240502 01:58:43.702308 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-16ae91433f10465c922183298a387caa on tablet 16ae91433f10465c922183298a387caa with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:43.702675 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 51462d8e7b414a169f3cfedb6b7ca585
I20240502 01:58:43.703119 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 51462d8e7b414a169f3cfedb6b7ca585 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:43.703253 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 51462d8e7b414a169f3cfedb6b7ca585 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:43.703313 29403 tablet_copy_client.cc:523] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.703424 29391 tablet_replica.cc:330] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:43.703532 29391 raft_consensus.cc:2229] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.703636 29391 raft_consensus.cc:2258] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.703773 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 51462d8e7b414a169f3cfedb6b7ca585 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:43 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:43.704238 29434 tablet_replica.cc:330] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:43.704385 29435 tablet_replica.cc:330] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:43.704401 29403 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:43.704535 29391 ts_tablet_manager.cc:1875] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.704567 29434 raft_consensus.cc:2229] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.704684 29434 raft_consensus.cc:2258] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.704843 29435 raft_consensus.cc:2229] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:43.705015 29435 raft_consensus.cc:2258] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.705525 29434 ts_tablet_manager.cc:1875] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.706634 29391 ts_tablet_manager.cc:1888] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.706701 29391 log.cc:1196] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/51462d8e7b414a169f3cfedb6b7ca585
I20240502 01:58:43.707515 29434 ts_tablet_manager.cc:1888] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.707949 29391 ts_tablet_manager.cc:1909] T 51462d8e7b414a169f3cfedb6b7ca585 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:43.708021 29435 ts_tablet_manager.cc:1875] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:43.708703 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 51462d8e7b414a169f3cfedb6b7ca585 (table rebalancer_dynamic_table_new_name [id=48af563d7bbb4a04b81926ea6791a88d]) successfully deleted
I20240502 01:58:43.709889 29435 ts_tablet_manager.cc:1888] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:43.709991 29435 log.cc:1196] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/51462d8e7b414a169f3cfedb6b7ca585
I20240502 01:58:43.710484 29435 ts_tablet_manager.cc:1909] T 51462d8e7b414a169f3cfedb6b7ca585 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:43.711004 29434 log.cc:1196] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/51462d8e7b414a169f3cfedb6b7ca585
I20240502 01:58:43.711052 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 51462d8e7b414a169f3cfedb6b7ca585 (table rebalancer_dynamic_table_new_name [id=48af563d7bbb4a04b81926ea6791a88d]) successfully deleted
I20240502 01:58:43.711339 29434 ts_tablet_manager.cc:1909] T 51462d8e7b414a169f3cfedb6b7ca585 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:43.711894 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 51462d8e7b414a169f3cfedb6b7ca585 (table rebalancer_dynamic_table_new_name [id=48af563d7bbb4a04b81926ea6791a88d]) successfully deleted
I20240502 01:58:43.720883 29424 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.721107 29424 tablet_bootstrap.cc:492] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:43.721238 29424 ts_tablet_manager.cc:1367] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.003s
I20240502 01:58:43.721751 27311 raft_consensus.cc:1203] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.722209 29424 raft_consensus.cc:348] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.722312 29424 raft_consensus.cc:729] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:43.722396 29424 consensus_queue.cc:260] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.722656 29424 ts_tablet_manager.cc:1398] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:58:43.724000 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-e720d8363bb049938798ab0b9aabc28d received from {username='slave'} at 127.17.243.8:44271
I20240502 01:58:43.724071 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session bb653087e8554c7591cf547dd969f3be-e720d8363bb049938798ab0b9aabc28d on tablet e720d8363bb049938798ab0b9aabc28d with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:43.730991 29403 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.731182 29403 tablet_bootstrap.cc:492] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:43.731312 29403 ts_tablet_manager.cc:1367] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.000s
I20240502 01:58:43.731643 29403 raft_consensus.cc:348] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.731753 29403 raft_consensus.cc:729] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:43.731846 29403 consensus_queue.cc:260] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:43.732106 29403 ts_tablet_manager.cc:1398] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:43.732528 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-8d5f6bcbe0a048eb85b9475348b155ef received from {username='slave'} at 127.17.243.7:55633
I20240502 01:58:43.732604 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-8d5f6bcbe0a048eb85b9475348b155ef on tablet 8d5f6bcbe0a048eb85b9475348b155ef with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:43.741628 29424 ts_tablet_manager.cc:902] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:43.741956 29424 tablet_copy_client.cc:308] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:43.742332 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 086c9b674a0543cab3c82349d151e373 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:45689)
I20240502 01:58:43.742408 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 086c9b674a0543cab3c82349d151e373 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:45689: session id = bb653087e8554c7591cf547dd969f3be-086c9b674a0543cab3c82349d151e373
I20240502 01:58:43.743018 25054 tablet_copy_source_session.cc:205] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.743650 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086c9b674a0543cab3c82349d151e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.745189 27578 raft_consensus.cc:1203] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.745919 29424 tablet_copy_client.cc:783] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:43.746073 29424 tablet_copy_client.cc:647] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:43.747380 29424 tablet_copy_client.cc:523] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:43.748456 29424 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:43.773545 29424 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:43.773757 29424 tablet_bootstrap.cc:492] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:43.773910 29424 ts_tablet_manager.cc:1367] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20240502 01:58:43.774319 29424 raft_consensus.cc:348] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.774420 29424 raft_consensus.cc:729] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:43.774506 29424 consensus_queue.cc:260] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:43.774755 29424 ts_tablet_manager.cc:1398] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:43.775188 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-086c9b674a0543cab3c82349d151e373 received from {username='slave'} at 127.17.243.8:45689
I20240502 01:58:43.775265 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session bb653087e8554c7591cf547dd969f3be-086c9b674a0543cab3c82349d151e373 on tablet 086c9b674a0543cab3c82349d151e373 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:43.781795 27867 raft_consensus.cc:482] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:43.781877 27867 raft_consensus.cc:504] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.782269 27867 leader_election.cc:288] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:43.782919 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73b38dc3e25640c0a6a8dd389ec67c1b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:43.783062 25032 raft_consensus.cc:2454] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:43.783363 27510 leader_election.cc:302] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:43.783596 27867 raft_consensus.cc:2781] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:43.783653 27867 raft_consensus.cc:482] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:43.783687 27867 raft_consensus.cc:3037] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.783753 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73b38dc3e25640c0a6a8dd389ec67c1b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:43.783870 24630 raft_consensus.cc:2454] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:43.784461 27867 raft_consensus.cc:504] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.784756 27867 leader_election.cc:288] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:43.785063 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73b38dc3e25640c0a6a8dd389ec67c1b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:43.785063 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73b38dc3e25640c0a6a8dd389ec67c1b" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:43.785164 24630 raft_consensus.cc:3037] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.785164 25032 raft_consensus.cc:3037] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.786094 25032 raft_consensus.cc:2454] T 73b38dc3e25640c0a6a8dd389ec67c1b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:43.786094 24630 raft_consensus.cc:2454] T 73b38dc3e25640c0a6a8dd389ec67c1b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:43.786371 27510 leader_election.cc:302] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:43.786643 27867 raft_consensus.cc:2781] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:43.786757 27867 raft_consensus.cc:686] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:43.786876 27867 consensus_queue.cc:237] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:43.788858 24485 catalog_manager.cc:5484] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.804963 24485 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:43.809537 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 0644a6a698934ba688a6ac8575dc64ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]), partition=
I20240502 01:58:43.809721 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 0644a6a698934ba688a6ac8575dc64ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]), partition=
I20240502 01:58:43.809764 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0644a6a698934ba688a6ac8575dc64ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.809954 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0644a6a698934ba688a6ac8575dc64ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.811707 29073 tablet_bootstrap.cc:492] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:43.811883 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 0644a6a698934ba688a6ac8575dc64ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]), partition=
I20240502 01:58:43.811960 29499 tablet_bootstrap.cc:492] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:43.812085 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0644a6a698934ba688a6ac8575dc64ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:43.812583 29073 tablet_bootstrap.cc:654] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.813169 29499 tablet_bootstrap.cc:654] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.814265 29499 tablet_bootstrap.cc:492] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:43.814486 29499 ts_tablet_manager.cc:1367] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:43.815013 29499 raft_consensus.cc:348] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.815109 29502 tablet_bootstrap.cc:492] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:43.815172 29499 raft_consensus.cc:374] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.815245 29499 raft_consensus.cc:729] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.815410 29499 consensus_queue.cc:260] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.815735 29499 ts_tablet_manager.cc:1398] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.815816 29502 tablet_bootstrap.cc:654] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:43.816763 29073 tablet_bootstrap.cc:492] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:43.816859 29073 ts_tablet_manager.cc:1367] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:43.817303 29073 raft_consensus.cc:348] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.817417 29073 raft_consensus.cc:374] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.817457 29073 raft_consensus.cc:729] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.817559 29073 consensus_queue.cc:260] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.817809 29073 ts_tablet_manager.cc:1398] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:43.818518 29502 tablet_bootstrap.cc:492] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:43.818650 29502 ts_tablet_manager.cc:1367] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:43.819941 29502 raft_consensus.cc:348] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.820108 29502 raft_consensus.cc:374] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:43.820190 29502 raft_consensus.cc:729] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:43.820317 29502 consensus_queue.cc:260] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.821331 29502 ts_tablet_manager.cc:1398] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:43.876951 28393 raft_consensus.cc:482] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:43.877049 28393 raft_consensus.cc:504] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.877480 28393 leader_election.cc:288] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:43.877784 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644a6a698934ba688a6ac8575dc64ce" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:43.877912 27446 raft_consensus.cc:2454] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:43.878175 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644a6a698934ba688a6ac8575dc64ce" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:43.878324 27311 raft_consensus.cc:2454] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:43.878384 24701 leader_election.cc:302] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:43.878734 28393 raft_consensus.cc:2781] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:43.878808 28393 raft_consensus.cc:482] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:43.878851 28393 raft_consensus.cc:3037] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.879621 28393 raft_consensus.cc:504] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.879959 28393 leader_election.cc:288] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:43.880234 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644a6a698934ba688a6ac8575dc64ce" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:43.880252 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0644a6a698934ba688a6ac8575dc64ce" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:43.880345 27446 raft_consensus.cc:3037] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.880350 27311 raft_consensus.cc:3037] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:43.881163 27446 raft_consensus.cc:2454] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:43.881310 27311 raft_consensus.cc:2454] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:43.881639 24701 leader_election.cc:302] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:43.883098 28393 raft_consensus.cc:2781] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:43.883191 28393 raft_consensus.cc:686] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:43.883306 28393 consensus_queue.cc:237] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:43.885139 24486 catalog_manager.cc:5484] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:43.928561 28396 raft_consensus.cc:1050] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:43.928988 28396 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:43.930500 25032 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.930665 27446 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.930950 27093 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.931530 27093 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.932119 24765 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:43.932521 27093 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:43.937284 27093 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.938707 27446 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.939522 24765 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.941639 24486 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:43.941805 25032 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.944787 24896 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:43.946082 24765 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:43.947314 27446 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:43.947542 29579 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:43.949131 29579 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:43.950199 27093 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.950567 24765 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.951702 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5ec1158a7414039b6f4bc855207c938 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:43.951766 27446 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:43.952522 24485 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:43.955753 25010 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ec1158a7414039b6f4bc855207c938 with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:43.955956 27740 tablet_replica.cc:330] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:43.956126 27740 raft_consensus.cc:2229] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:43.956264 27740 raft_consensus.cc:2258] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:43.956980 27740 ts_tablet_manager.cc:1875] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:43.961076 27740 ts_tablet_manager.cc:1888] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:43.961153 27740 log.cc:1196] T d5ec1158a7414039b6f4bc855207c938 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5ec1158a7414039b6f4bc855207c938
I20240502 01:58:43.963645 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet d5ec1158a7414039b6f4bc855207c938 (table rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]) successfully deleted
I20240502 01:58:43.963928 27424 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.966967 27155 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.967208 27290 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:43.967813 24610 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:43.980087 24743 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:58:43.991410 27557 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:43.995728 27867 raft_consensus.cc:1050] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be: attempting to promote NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 to VOTER
I20240502 01:58:43.996110 27311 raft_consensus.cc:1203] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:43.996210 27867 consensus_queue.cc:237] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:43.997839 25032 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:43.998245 28874 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:43.998378 27177 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:43.998742 24630 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:43.999081 27865 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:43.999428 27865 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:44.000051 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:44.002640 25010 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:58:44.004877 29102 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.006302 29595 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.009531 24630 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.010486 25032 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.012768 24485 catalog_manager.cc:5484] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.013715 24877 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:58:44.014487 27176 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.019044 29174 ts_tablet_manager.cc:902] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:44.019068 27578 consensus_queue.cc:237] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:44.019438 29174 tablet_copy_client.cc:308] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:44.019871 27311 raft_consensus.cc:1203] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.020759 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 48085cebf94e4f549e6174a88acdcc59 from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:44697)
I20240502 01:58:44.020846 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 48085cebf94e4f549e6174a88acdcc59 from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:44697: session id = 1049b4c8783b48f39be22bd113ad349f-48085cebf94e4f549e6174a88acdcc59
I20240502 01:58:44.021736 24918 tablet_copy_source_session.cc:205] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.022604 25032 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.023020 27865 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.023146 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48085cebf94e4f549e6174a88acdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.025177 29174 tablet_copy_client.cc:783] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.025310 29174 tablet_copy_client.cc:647] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.026262 24630 raft_consensus.cc:1261] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.026433 27446 raft_consensus.cc:1261] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.026759 27311 raft_consensus.cc:1261] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.026882 28568 consensus_queue.cc:1035] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.027194 28568 consensus_queue.cc:1035] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.029079 28874 consensus_queue.cc:1035] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.029503 24478 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]). Expected version 1 got 0
I20240502 01:58:44.030023 29605 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.030625 27445 raft_consensus.cc:1203] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.033056 24483 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]). Expected version 1 got 0
I20240502 01:58:44.035135 27865 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.036796 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 823fe241dc1b40cca121a22d669e4226 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.036720 24630 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.036795 25032 raft_consensus.cc:2932] T 823fe241dc1b40cca121a22d669e4226 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:44.039161 24478 catalog_manager.cc:5484] T 823fe241dc1b40cca121a22d669e4226 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.044499 24478 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]). Expected version 1 got 0
I20240502 01:58:44.045159 24485 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]). Expected version 1 got 0
I20240502 01:58:44.046219 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 823fe241dc1b40cca121a22d669e4226 with delete_type TABLET_DATA_TOMBSTONED (TS 9dd619a595e94004ac0f7f5a9054e7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:44.046355 29247 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:44.046411 29391 tablet_replica.cc:330] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:44.046525 29391 raft_consensus.cc:2229] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.046625 29391 raft_consensus.cc:2258] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.046638 29247 tablet.cc:1620] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:44.048743 29391 ts_tablet_manager.cc:1875] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.050541 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44] alter complete (version 1)
I20240502 01:58:44.050889 29391 ts_tablet_manager.cc:1888] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:44.050962 29391 log.cc:1196] T 823fe241dc1b40cca121a22d669e4226 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/823fe241dc1b40cca121a22d669e4226
I20240502 01:58:44.051486 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 823fe241dc1b40cca121a22d669e4226 (table rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]) successfully deleted
I20240502 01:58:44.051788 29610 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:44.052327 29174 tablet_copy_client.cc:523] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.052656 29610 tablet.cc:1620] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:44.053582 29174 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:44.054330 24485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 01:58:44.054451 24485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 01:58:44.054556 28851 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:44.054797 28851 tablet.cc:1620] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:44.056270 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 1a383190824e4611a9e279b05bfe4e2a
I20240502 01:58:44.056507 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet a094659d539e4c4f9861e356168c5894
I20240502 01:58:44.056715 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 752d360cfdd64846b6cd37c9b859d7a8
I20240502 01:58:44.057497 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:44.057492 24478 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44]). Expected version 1 got 0
I20240502 01:58:44.057643 29435 tablet_replica.cc:330] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:44.057713 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet a094659d539e4c4f9861e356168c5894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:44.057754 29435 raft_consensus.cc:2229] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.057873 29435 raft_consensus.cc:2258] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.057876 29434 tablet_replica.cc:330] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:44.058034 29434 raft_consensus.cc:2229] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.058151 29434 raft_consensus.cc:2258] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.058158 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a383190824e4611a9e279b05bfe4e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.058336 27864 tablet_replica.cc:330] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.058322 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a383190824e4611a9e279b05bfe4e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:44.058414 29435 ts_tablet_manager.cc:1875] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.058472 27864 raft_consensus.cc:2229] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.058580 29434 ts_tablet_manager.cc:1875] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.058645 27864 raft_consensus.cc:2258] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.059072 27864 ts_tablet_manager.cc:1875] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.060364 29435 ts_tablet_manager.cc:1888] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.060431 29435 log.cc:1196] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/752d360cfdd64846b6cd37c9b859d7a8
I20240502 01:58:44.060427 29434 ts_tablet_manager.cc:1888] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.060487 29434 log.cc:1196] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/a094659d539e4c4f9861e356168c5894
I20240502 01:58:44.060710 29435 ts_tablet_manager.cc:1909] T 752d360cfdd64846b6cd37c9b859d7a8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:44.060731 29434 ts_tablet_manager.cc:1909] T a094659d539e4c4f9861e356168c5894 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:44.061094 27864 ts_tablet_manager.cc:1888] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.061151 27864 log.cc:1196] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a383190824e4611a9e279b05bfe4e2a
I20240502 01:58:44.061261 25010 tablet_service.cc:1514] Processing DeleteTablet for tablet a094659d539e4c4f9861e356168c5894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:44.061306 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.061371 27864 ts_tablet_manager.cc:1909] T 1a383190824e4611a9e279b05bfe4e2a P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:44.061434 29390 tablet_replica.cc:330] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:44.061529 28540 tablet_replica.cc:330] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.061549 29390 raft_consensus.cc:2229] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.061643 28540 raft_consensus.cc:2229] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.061663 29390 raft_consensus.cc:2258] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.061839 28540 raft_consensus.cc:2258] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.061918 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet a094659d539e4c4f9861e356168c5894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.062090 24875 tablet_service.cc:1514] Processing DeleteTablet for tablet 752d360cfdd64846b6cd37c9b859d7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:44.062162 29390 ts_tablet_manager.cc:1875] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.062336 28540 ts_tablet_manager.cc:1875] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.062664 27740 tablet_replica.cc:330] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:44.062777 27740 raft_consensus.cc:2229] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.062920 27740 raft_consensus.cc:2258] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.063362 27740 ts_tablet_manager.cc:1875] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.064009 29390 ts_tablet_manager.cc:1888] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.064067 29390 log.cc:1196] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/1a383190824e4611a9e279b05bfe4e2a
I20240502 01:58:44.064296 29390 ts_tablet_manager.cc:1909] T 1a383190824e4611a9e279b05bfe4e2a P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:44.064288 28540 ts_tablet_manager.cc:1888] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.064347 28540 log.cc:1196] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/752d360cfdd64846b6cd37c9b859d7a8
I20240502 01:58:44.064579 28540 ts_tablet_manager.cc:1909] T 752d360cfdd64846b6cd37c9b859d7a8 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:44.064968 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a383190824e4611a9e279b05bfe4e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:44.065068 28540 tablet_replica.cc:330] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.065171 28540 raft_consensus.cc:2229] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.065326 28540 raft_consensus.cc:2258] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.065407 27740 ts_tablet_manager.cc:1888] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.065477 27740 log.cc:1196] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/a094659d539e4c4f9861e356168c5894
I20240502 01:58:44.065748 27740 ts_tablet_manager.cc:1909] T a094659d539e4c4f9861e356168c5894 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:44.065793 28540 ts_tablet_manager.cc:1875] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.066064 28027 tablet_replica.cc:330] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.066258 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 752d360cfdd64846b6cd37c9b859d7a8 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.067245 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet a094659d539e4c4f9861e356168c5894 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.067293 28027 raft_consensus.cc:2229] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.067456 28027 raft_consensus.cc:2258] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.067612 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 752d360cfdd64846b6cd37c9b859d7a8 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.067726 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 1a383190824e4611a9e279b05bfe4e2a (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.067817 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 1a383190824e4611a9e279b05bfe4e2a (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.067924 28027 ts_tablet_manager.cc:1875] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.068336 28540 ts_tablet_manager.cc:1888] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.068392 28540 log.cc:1196] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/a094659d539e4c4f9861e356168c5894
I20240502 01:58:44.068663 28540 ts_tablet_manager.cc:1909] T a094659d539e4c4f9861e356168c5894 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:44.069309 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet a094659d539e4c4f9861e356168c5894 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.069591 28027 ts_tablet_manager.cc:1888] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.069638 28027 log.cc:1196] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/752d360cfdd64846b6cd37c9b859d7a8
I20240502 01:58:44.069787 28027 ts_tablet_manager.cc:1909] T 752d360cfdd64846b6cd37c9b859d7a8 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:44.070271 29595 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.070869 28027 tablet_replica.cc:330] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.070971 28027 raft_consensus.cc:2229] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.070963 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 752d360cfdd64846b6cd37c9b859d7a8 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.071107 28027 raft_consensus.cc:2258] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.072541 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet a094659d539e4c4f9861e356168c5894 (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.075841 28027 ts_tablet_manager.cc:1875] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.077128 29595 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.080969 28027 ts_tablet_manager.cc:1888] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240502 01:58:44.081089 28027 log.cc:1196] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/1a383190824e4611a9e279b05bfe4e2a
I20240502 01:58:44.081380 28027 ts_tablet_manager.cc:1909] T 1a383190824e4611a9e279b05bfe4e2a P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:44.082054 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 1a383190824e4611a9e279b05bfe4e2a (table rebalance_test_table_16 [id=8476d69563ff4f0c8a7d4cbd14c1fee3]) successfully deleted
I20240502 01:58:44.091866 24765 consensus_queue.cc:237] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.093070 24630 raft_consensus.cc:1261] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.093138 25032 raft_consensus.cc:1261] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.093420 28568 consensus_queue.cc:1035] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.093575 28404 consensus_queue.cc:1035] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:44.093724 24700 consensus_peers.cc:485] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 751feb4b2419463c81f3571f6eb0fa42. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.093813 29174 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.094076 29174 tablet_bootstrap.cc:492] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:44.094288 29174 ts_tablet_manager.cc:1367] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.003s
I20240502 01:58:44.094750 29174 raft_consensus.cc:348] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:44.094930 29174 raft_consensus.cc:729] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:44.095084 29174 consensus_queue.cc:260] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:44.095376 29174 ts_tablet_manager.cc:1398] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:44.095835 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-48085cebf94e4f549e6174a88acdcc59 received from {username='slave'} at 127.17.243.6:44697
I20240502 01:58:44.095904 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-48085cebf94e4f549e6174a88acdcc59 on tablet 48085cebf94e4f549e6174a88acdcc59 with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:44.095826 28404 raft_consensus.cc:2932] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.096161 25032 raft_consensus.cc:2932] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.098111 24630 raft_consensus.cc:2932] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.098551 24478 catalog_manager.cc:5484] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.108785 28568 raft_consensus.cc:1050] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:44.109122 28568 consensus_queue.cc:237] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.110373 24630 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.110427 27578 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.110492 24896 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.110847 29629 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.111255 27446 consensus_queue.cc:237] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:44.111373 29629 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.111658 29629 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.112713 27311 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.113067 28447 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:58:44.113222 27380 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85e6a0806654abd81ca5661a2d019ce. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.113551 27576 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.114192 28447 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.114532 28447 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.114889 27311 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.117357 24478 catalog_manager.cc:5484] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.117588 27576 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.120409 27890 raft_consensus.cc:1050] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:44.120760 27890 consensus_queue.cc:237] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.121448 28568 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.122138 27177 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:44.122498 27894 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:44.123005 24630 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:44.123250 27446 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:44.123574 27894 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:44.124171 24630 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.125703 27894 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:44.126358 24478 catalog_manager.cc:5484] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.126403 24896 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.127713 27578 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.129336 24765 consensus_queue.cc:237] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.130383 24630 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.132285 27578 raft_consensus.cc:1261] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.132874 28534 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.134452 27446 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.134754 27177 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.134877 29629 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.136310 24478 catalog_manager.cc:5484] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.137269 29629 consensus_queue.cc:1035] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.138247 28404 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.138681 24632 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.140012 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ac6a4310330499483182c93827343c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.141033 24483 catalog_manager.cc:5484] T 4ac6a4310330499483182c93827343c0 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.142442 27446 raft_consensus.cc:1203] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.142851 27578 raft_consensus.cc:2932] T 4ac6a4310330499483182c93827343c0 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.144788 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac6a4310330499483182c93827343c0 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:44.144961 28027 tablet_replica.cc:330] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.145072 28027 raft_consensus.cc:2229] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.145210 28027 raft_consensus.cc:2258] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.145826 28027 ts_tablet_manager.cc:1875] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.146201 28568 raft_consensus.cc:1050] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:44.146508 28568 consensus_queue.cc:237] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.147835 28027 ts_tablet_manager.cc:1888] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.147897 28027 log.cc:1196] T 4ac6a4310330499483182c93827343c0 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/4ac6a4310330499483182c93827343c0
I20240502 01:58:44.148021 25032 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.148154 24630 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.148358 29629 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.149335 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 4ac6a4310330499483182c93827343c0 (table rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]) successfully deleted
I20240502 01:58:44.150548 27578 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.150919 29629 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.150985 25032 consensus_queue.cc:237] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.151523 24896 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.151917 29629 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.152341 27446 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.152530 24478 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:44.152798 27017 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.153468 24630 raft_consensus.cc:1261] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.153983 27890 consensus_queue.cc:1035] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.154222 29629 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.155478 27894 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.155920 24630 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.156386 25032 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.157974 24896 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.159029 27446 raft_consensus.cc:2932] T 6560bc797c024a90aa87515d0ef2c4c6 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.159152 24483 catalog_manager.cc:5484] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.161355 24485 catalog_manager.cc:5484] T 6560bc797c024a90aa87515d0ef2c4c6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.161911 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6560bc797c024a90aa87515d0ef2c4c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.162129 28534 raft_consensus.cc:1050] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:44.162127 29610 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:44.162412 29610 tablet.cc:1620] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:44.162456 28534 consensus_queue.cc:237] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.163420 28851 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:44.163671 28851 tablet.cc:1620] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:44.163820 24630 raft_consensus.cc:1261] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.164167 27894 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.165410 27312 raft_consensus.cc:1203] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.165993 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6560bc797c024a90aa87515d0ef2c4c6 with delete_type TABLET_DATA_TOMBSTONED (TS 9dd619a595e94004ac0f7f5a9054e7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:44.166185 29391 tablet_replica.cc:330] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:44.166302 29391 raft_consensus.cc:2229] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.166424 29391 raft_consensus.cc:2258] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.167037 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=dc048ee726944c97a23be31a4e17fd44] alter complete (version 2)
I20240502 01:58:44.168057 27578 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.169229 24765 raft_consensus.cc:1261] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.169229 27311 raft_consensus.cc:1261] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.169582 27890 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.171361 24765 consensus_queue.cc:237] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.171721 29247 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:44.171999 29247 tablet.cc:1620] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:44.172456 27578 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.172878 28404 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.173804 27894 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.174487 28404 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.174890 27894 consensus_queue.cc:1035] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.174865 27578 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.175662 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebe8db639839433d967583371351eacb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.175845 24765 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.176643 27311 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.177086 25032 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.177695 29391 ts_tablet_manager.cc:1875] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.177665 24479 catalog_manager.cc:5484] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.177752 28393 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.178648 25032 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.178961 24478 catalog_manager.cc:5484] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.179718 29391 ts_tablet_manager.cc:1888] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:44.179785 29391 log.cc:1196] T 6560bc797c024a90aa87515d0ef2c4c6 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/6560bc797c024a90aa87515d0ef2c4c6
I20240502 01:58:44.180167 24630 raft_consensus.cc:2932] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.182319 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe8db639839433d967583371351eacb with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:44.182538 28027 tablet_replica.cc:330] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.182631 28027 raft_consensus.cc:2229] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.182745 28027 raft_consensus.cc:2258] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.183264 28027 ts_tablet_manager.cc:1875] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.184352 27578 raft_consensus.cc:1203] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.184949 28027 ts_tablet_manager.cc:1888] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.184994 28027 log.cc:1196] T ebe8db639839433d967583371351eacb P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/ebe8db639839433d967583371351eacb
I20240502 01:58:44.185349 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet ebe8db639839433d967583371351eacb (table rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]) successfully deleted
I20240502 01:58:44.186367 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 6560bc797c024a90aa87515d0ef2c4c6 (table rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]) successfully deleted
I20240502 01:58:44.189862 25032 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.191087 27578 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:58:44.191109 24965 consensus_peers.cc:485] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 488816b0d5d542bcb5fa12ff7e70fd40. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.191433 29648 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:44.191486 24765 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:44.191949 29648 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:44.193111 27894 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.193742 24765 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.195642 24486 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.197572 27578 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.202262 27446 raft_consensus.cc:1203] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.235635 27576 raft_consensus.cc:1203] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:44.243549 28393 raft_consensus.cc:1050] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:44.243985 28393 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:44.245306 24896 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.245370 27177 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.245625 25032 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.245775 29629 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.246188 29629 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.246817 28404 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.248113 28393 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.248908 25032 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.250442 27177 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.251029 24896 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.253201 24479 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.256247 24765 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:44.257171 25032 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.257171 27177 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.257509 28568 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.257639 28404 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.259109 28404 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.259485 25032 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.259865 27177 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.261366 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63312ec15b034e47bc2dd484b57d4648 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.262167 24483 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:44.264542 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 63312ec15b034e47bc2dd484b57d4648 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:44.264734 28027 tablet_replica.cc:330] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.264868 28027 raft_consensus.cc:2229] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.265005 28027 raft_consensus.cc:2258] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.265584 28027 ts_tablet_manager.cc:1875] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.267560 28027 ts_tablet_manager.cc:1888] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.267625 28027 log.cc:1196] T 63312ec15b034e47bc2dd484b57d4648 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/63312ec15b034e47bc2dd484b57d4648
I20240502 01:58:44.268031 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 63312ec15b034e47bc2dd484b57d4648 (table rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]) successfully deleted
I20240502 01:58:44.269237 24478 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:44.274164 29610 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:44.275071 29247 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:44.276775 28851 tablet.cc:1722] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:44.280977 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=dc048ee726944c97a23be31a4e17fd44] alter complete (version 3)
I20240502 01:58:44.314903 29595 consensus_queue.cc:1035] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.318989 27867 consensus_queue.cc:1035] T 73b38dc3e25640c0a6a8dd389ec67c1b P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.324400 27446 consensus_queue.cc:237] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:44.325627 27311 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.325656 27177 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:44.325836 27380 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c09e8f421d4ae2adf7c9b63121943c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.326032 29244 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.326205 28447 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.327793 28315 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.328147 27311 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.328253 27177 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.330372 24478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:44.330466 24483 catalog_manager.cc:5484] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:44.336141 27155 tablet_service.cc:1467] Processing CreateTablet for tablet dad6070e3df4409a9e684a435e780799 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.336405 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad6070e3df4409a9e684a435e780799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.336722 24610 tablet_service.cc:1467] Processing CreateTablet for tablet cccdcec2dc2441b29fe7c7f261478bf1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.336783 25010 tablet_service.cc:1467] Processing CreateTablet for tablet dad6070e3df4409a9e684a435e780799 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.336973 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdcec2dc2441b29fe7c7f261478bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.337004 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad6070e3df4409a9e684a435e780799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.338120 29770 tablet_bootstrap.cc:492] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:44.338546 27558 tablet_service.cc:1467] Processing CreateTablet for tablet cccdcec2dc2441b29fe7c7f261478bf1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.338798 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdcec2dc2441b29fe7c7f261478bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.339282 29772 tablet_bootstrap.cc:492] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:44.339248 27557 tablet_service.cc:1467] Processing CreateTablet for tablet dad6070e3df4409a9e684a435e780799 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.339370 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c460a54024915a2944c2f44a10b05 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.339461 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad6070e3df4409a9e684a435e780799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.339589 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c460a54024915a2944c2f44a10b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.340183 29772 tablet_bootstrap.cc:654] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.340726 29771 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:44.341425 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c460a54024915a2944c2f44a10b05 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.341521 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c460a54024915a2944c2f44a10b05 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.341619 25012 tablet_service.cc:1467] Processing CreateTablet for tablet cccdcec2dc2441b29fe7c7f261478bf1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.341784 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c460a54024915a2944c2f44a10b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.341794 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdcec2dc2441b29fe7c7f261478bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.342170 29771 tablet_bootstrap.cc:654] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.343115 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c460a54024915a2944c2f44a10b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.343495 29771 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:44.343565 29771 ts_tablet_manager.cc:1367] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:44.343924 29771 raft_consensus.cc:348] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.344027 29771 raft_consensus.cc:374] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.344056 29771 raft_consensus.cc:729] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.344127 29777 tablet_bootstrap.cc:492] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:44.344170 29771 consensus_queue.cc:260] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.344437 29771 ts_tablet_manager.cc:1398] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.345345 29777 tablet_bootstrap.cc:654] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.345983 29770 tablet_bootstrap.cc:654] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.346307 29777 tablet_bootstrap.cc:492] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:44.346392 29777 ts_tablet_manager.cc:1367] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:44.346401 29774 tablet_bootstrap.cc:492] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:44.346637 29773 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:44.346793 29777 raft_consensus.cc:348] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.346899 29777 raft_consensus.cc:374] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.346925 29777 raft_consensus.cc:729] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.347040 29777 consensus_queue.cc:260] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.347313 29777 ts_tablet_manager.cc:1398] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:44.347478 29778 tablet_bootstrap.cc:492] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:44.347522 29774 tablet_bootstrap.cc:654] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.347963 29773 tablet_bootstrap.cc:654] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.350005 29774 tablet_bootstrap.cc:492] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:44.350172 29774 ts_tablet_manager.cc:1367] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:44.350349 29778 tablet_bootstrap.cc:654] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.350585 29774 raft_consensus.cc:348] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.350749 29774 raft_consensus.cc:374] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.350797 29774 raft_consensus.cc:729] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.350899 29774 consensus_queue.cc:260] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.351341 29773 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:44.351385 25032 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.352035 29770 tablet_bootstrap.cc:492] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:44.352165 29770 ts_tablet_manager.cc:1367] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 01:58:44.352275 29773 ts_tablet_manager.cc:1367] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:58:44.352644 29770 raft_consensus.cc:348] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.352785 29770 raft_consensus.cc:374] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.352821 29770 raft_consensus.cc:729] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.352926 29770 consensus_queue.cc:260] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
W20240502 01:58:44.353368 24965 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26b7501639064035a570b3b4c1cf083f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.353368 24896 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:44.353761 29648 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:44.353855 29774 ts_tablet_manager.cc:1398] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:44.353902 27311 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:44.354682 29770 ts_tablet_manager.cc:1398] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:44.355006 29648 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:44.355129 29658 raft_consensus.cc:482] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:44.355201 29658 raft_consensus.cc:504] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.355366 29773 raft_consensus.cc:348] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.355454 29773 raft_consensus.cc:374] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.355484 29773 raft_consensus.cc:729] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.355577 29658 leader_election.cc:288] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:44.355922 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdcec2dc2441b29fe7c7f261478bf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:44.356218 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdcec2dc2441b29fe7c7f261478bf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
W20240502 01:58:44.356319 24564 leader_election.cc:341] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:44.356483 29773 consensus_queue.cc:260] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.356774 27576 raft_consensus.cc:2454] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:44.357228 29773 ts_tablet_manager.cc:1398] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:44.357393 29773 tablet_bootstrap.cc:492] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:44.358052 27894 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.358530 29772 tablet_bootstrap.cc:492] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:44.358589 29772 ts_tablet_manager.cc:1367] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240502 01:58:44.358953 29772 raft_consensus.cc:348] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.359102 29772 raft_consensus.cc:374] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.359174 29772 raft_consensus.cc:729] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.359318 29772 consensus_queue.cc:260] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.357854 24567 leader_election.cc:302] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters: e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:58:44.359624 29772 ts_tablet_manager.cc:1398] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.359807 29772 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:44.360169 29658 raft_consensus.cc:2781] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:44.360234 29658 raft_consensus.cc:482] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:44.360283 29658 raft_consensus.cc:3037] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.360530 29772 tablet_bootstrap.cc:654] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.361140 24479 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.361274 29773 tablet_bootstrap.cc:654] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.361263 29658 raft_consensus.cc:504] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.361459 29772 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:44.361526 29772 ts_tablet_manager.cc:1367] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:44.361582 29658 leader_election.cc:288] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:44.361914 29772 raft_consensus.cc:348] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.362687 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdcec2dc2441b29fe7c7f261478bf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:44.362303 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccdcec2dc2441b29fe7c7f261478bf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:44.362771 27576 raft_consensus.cc:3037] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.362864 29772 raft_consensus.cc:374] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.362906 29772 raft_consensus.cc:729] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.363304 29778 tablet_bootstrap.cc:492] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:44.363382 29778 ts_tablet_manager.cc:1367] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:44.363642 27576 raft_consensus.cc:2454] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:44.363826 29778 raft_consensus.cc:348] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.364006 29778 raft_consensus.cc:374] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.364018 24567 leader_election.cc:302] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:44.364298 28799 raft_consensus.cc:2781] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:44.364382 28799 raft_consensus.cc:686] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:44.364502 28799 consensus_queue.cc:237] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.364534 27311 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.364684 29778 raft_consensus.cc:729] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.364652 24896 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.365031 29778 consensus_queue.cc:260] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.366043 29778 ts_tablet_manager.cc:1398] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:44.367668 29773 tablet_bootstrap.cc:492] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:44.367693 24483 catalog_manager.cc:5484] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.368388 29773 ts_tablet_manager.cc:1367] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:58:44.363036 29772 consensus_queue.cc:260] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.368557 25032 raft_consensus.cc:3037] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.368958 29773 raft_consensus.cc:348] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.369102 29773 raft_consensus.cc:374] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.369170 29773 raft_consensus.cc:729] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.369349 29773 consensus_queue.cc:260] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.369468 25032 raft_consensus.cc:2454] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:44.370277 29773 ts_tablet_manager.cc:1398] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:58:44.370604 29772 ts_tablet_manager.cc:1398] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240502 01:58:44.376688 24896 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:44.377961 24630 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.378361 24765 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.378337 29573 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.379630 29573 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.380154 27093 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:44.380576 24630 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:44.382004 24765 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
W20240502 01:58:44.382612 24833 consensus_peers.cc:485] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44be529dec974eac8419efe9c19b30bf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.383004 24479 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:44.385193 24478 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:44.385288 24478 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:44.387403 24478 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 0644a6a698934ba688a6ac8575dc64ce
I20240502 01:58:44.388017 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644a6a698934ba688a6ac8575dc64ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:44.388178 29434 tablet_replica.cc:330] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:44.388294 29434 raft_consensus.cc:2229] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.388397 29434 raft_consensus.cc:2258] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.388509 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644a6a698934ba688a6ac8575dc64ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.388679 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0644a6a698934ba688a6ac8575dc64ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:44.388828 29435 tablet_replica.cc:330] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:44.388828 27864 tablet_replica.cc:330] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.388938 27864 raft_consensus.cc:2229] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.388938 29435 raft_consensus.cc:2229] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.389045 29435 raft_consensus.cc:2258] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.389060 29434 ts_tablet_manager.cc:1875] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.389102 27864 raft_consensus.cc:2258] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.389590 29435 ts_tablet_manager.cc:1875] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.389971 27864 ts_tablet_manager.cc:1875] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.390868 29434 ts_tablet_manager.cc:1888] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:44.390923 29434 log.cc:1196] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/0644a6a698934ba688a6ac8575dc64ce
I20240502 01:58:44.391088 29434 ts_tablet_manager.cc:1909] T 0644a6a698934ba688a6ac8575dc64ce P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:44.391669 29435 ts_tablet_manager.cc:1888] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:44.391729 29435 log.cc:1196] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/0644a6a698934ba688a6ac8575dc64ce
I20240502 01:58:44.391847 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table_new_name [id=dc048ee726944c97a23be31a4e17fd44]) successfully deleted
I20240502 01:58:44.391948 27864 ts_tablet_manager.cc:1888] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:44.391966 29435 ts_tablet_manager.cc:1909] T 0644a6a698934ba688a6ac8575dc64ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:44.392007 27864 log.cc:1196] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/0644a6a698934ba688a6ac8575dc64ce
I20240502 01:58:44.392212 27864 ts_tablet_manager.cc:1909] T 0644a6a698934ba688a6ac8575dc64ce P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:44.393306 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table_new_name [id=dc048ee726944c97a23be31a4e17fd44]) successfully deleted
I20240502 01:58:44.393604 29573 raft_consensus.cc:1050] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:44.393675 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 0644a6a698934ba688a6ac8575dc64ce (table rebalancer_dynamic_table_new_name [id=dc048ee726944c97a23be31a4e17fd44]) successfully deleted
I20240502 01:58:44.393999 29573 consensus_queue.cc:237] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.395304 27311 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.395224 24765 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.395655 27093 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.395804 25032 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.395910 29578 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.396427 29578 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.398322 29578 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.398648 25032 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.399396 29244 raft_consensus.cc:482] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:44.399462 29244 raft_consensus.cc:504] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.399828 29244 leader_election.cc:288] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:44.401011 27312 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.401041 24478 catalog_manager.cc:5484] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.401707 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c460a54024915a2944c2f44a10b05" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:44.401808 24896 raft_consensus.cc:2454] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:44.402117 27380 leader_election.cc:302] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:44.402262 29244 raft_consensus.cc:2781] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:44.402305 29244 raft_consensus.cc:482] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:44.402277 24897 consensus_queue.cc:237] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:44.402351 29244 raft_consensus.cc:3037] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.403213 29244 raft_consensus.cc:504] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.403548 29244 leader_election.cc:288] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:44.403558 24763 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.403666 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c460a54024915a2944c2f44a10b05" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:44.403761 27311 raft_consensus.cc:2454] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:44.403789 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c460a54024915a2944c2f44a10b05" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:44.403887 27312 raft_consensus.cc:3037] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.403558 24765 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.404222 25032 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.404656 27312 raft_consensus.cc:2454] T 3a5c460a54024915a2944c2f44a10b05 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:44.404841 27093 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.405411 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c460a54024915a2944c2f44a10b05" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:44.405519 24897 raft_consensus.cc:3037] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.405774 27377 leader_election.cc:302] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
W20240502 01:58:44.406302 24833 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89580549c13d4c1ebdb2a5d24da2a01e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.406433 24897 raft_consensus.cc:2454] T 3a5c460a54024915a2944c2f44a10b05 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:44.406415 29574 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.406984 29244 raft_consensus.cc:2781] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:44.407084 29244 raft_consensus.cc:686] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:44.407207 29244 consensus_queue.cc:237] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:44.409001 24483 catalog_manager.cc:5484] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.409946 29573 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.410311 24763 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.411031 24897 consensus_queue.cc:237] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.412302 24763 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.412719 27312 raft_consensus.cc:1261] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.412662 24486 catalog_manager.cc:5484] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.412717 25032 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.413807 27093 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.414419 27093 consensus_queue.cc:1035] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.419631 29574 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.421864 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d0cc9aba3804968a6dce97682f6b38e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.422032 29578 raft_consensus.cc:1050] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:44.422302 29578 consensus_queue.cc:237] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.423583 24630 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.424073 24763 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.424180 27312 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.424244 27311 raft_consensus.cc:2932] T 7d0cc9aba3804968a6dce97682f6b38e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.424499 24765 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.425900 29578 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.426288 29578 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.426638 24483 catalog_manager.cc:5484] T 7d0cc9aba3804968a6dce97682f6b38e P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 5 to 6, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.426694 29579 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.428391 27312 raft_consensus.cc:1203] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:44.428773 29648 raft_consensus.cc:482] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:44.428853 29648 raft_consensus.cc:504] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.429198 29648 leader_election.cc:288] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:44.429572 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad6070e3df4409a9e684a435e780799" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:44.429693 27576 raft_consensus.cc:2454] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:44.430164 24966 leader_election.cc:302] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:44.430176 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad6070e3df4409a9e684a435e780799" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:44.430284 27177 raft_consensus.cc:2454] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:44.430539 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0cc9aba3804968a6dce97682f6b38e with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:44.430774 27740 tablet_replica.cc:330] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:44.430902 27740 raft_consensus.cc:2229] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.431041 27740 raft_consensus.cc:2258] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.431291 29648 raft_consensus.cc:2781] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:44.431349 29648 raft_consensus.cc:482] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:44.431387 29648 raft_consensus.cc:3037] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.431639 27740 ts_tablet_manager.cc:1875] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.432058 29648 raft_consensus.cc:504] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.432319 29648 leader_election.cc:288] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:44.432662 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad6070e3df4409a9e684a435e780799" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:44.432662 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad6070e3df4409a9e684a435e780799" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:44.432751 27576 raft_consensus.cc:3037] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.432751 27177 raft_consensus.cc:3037] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.433530 27740 ts_tablet_manager.cc:1888] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.433583 27740 log.cc:1196] T 7d0cc9aba3804968a6dce97682f6b38e P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d0cc9aba3804968a6dce97682f6b38e
I20240502 01:58:44.433612 27177 raft_consensus.cc:2454] T dad6070e3df4409a9e684a435e780799 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:44.433612 27576 raft_consensus.cc:2454] T dad6070e3df4409a9e684a435e780799 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:44.433869 24965 leader_election.cc:302] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:44.434182 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 7d0cc9aba3804968a6dce97682f6b38e (table rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]) successfully deleted
I20240502 01:58:44.435160 29648 raft_consensus.cc:2781] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:44.435242 29648 raft_consensus.cc:686] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:44.435350 29648 consensus_queue.cc:237] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.437229 24479 catalog_manager.cc:5484] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.438047 29574 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.439600 27311 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.439651 24763 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.443037 24479 catalog_manager.cc:5484] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.447036 24897 consensus_queue.cc:237] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.447238 29648 raft_consensus.cc:1050] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:44.447582 29648 consensus_queue.cc:237] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.447963 24630 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.448786 24632 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.449110 27446 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.449290 28534 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.449647 27311 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.450387 24897 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.450418 29573 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.450625 24763 raft_consensus.cc:1261] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.450682 28534 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.450992 28534 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.451489 29573 consensus_queue.cc:1035] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.454619 29573 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.456802 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e47667fc8fa49e6b18ee3df06cda00a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.456758 27311 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.456840 24763 raft_consensus.cc:2932] T 0e47667fc8fa49e6b18ee3df06cda00a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.459120 27017 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.459488 24483 catalog_manager.cc:5484] T 0e47667fc8fa49e6b18ee3df06cda00a P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.461318 27446 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.462235 24479 catalog_manager.cc:5484] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.463354 24897 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.466032 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e47667fc8fa49e6b18ee3df06cda00a with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.466321 28540 tablet_replica.cc:330] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.466470 28540 raft_consensus.cc:2229] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.466627 28540 raft_consensus.cc:2258] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.468581 25032 consensus_queue.cc:237] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.469539 28540 ts_tablet_manager.cc:1875] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.470113 24897 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.470124 27446 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.470571 27894 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.470829 29648 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.471665 28540 ts_tablet_manager.cc:1888] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.472326 27894 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.473131 27446 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.473624 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b8667a49f1a44339122040b0122be0d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.475292 24897 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.475476 24479 catalog_manager.cc:5484] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.479511 28540 log.cc:1196] T 0e47667fc8fa49e6b18ee3df06cda00a P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e47667fc8fa49e6b18ee3df06cda00a
I20240502 01:58:44.479539 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8667a49f1a44339122040b0122be0d with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.479947 28540 tablet_replica.cc:330] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.480058 28540 raft_consensus.cc:2229] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.480054 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 0e47667fc8fa49e6b18ee3df06cda00a (table rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]) successfully deleted
I20240502 01:58:44.480172 28540 pending_rounds.cc:62] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Trying to abort 1 pending ops.
I20240502 01:58:44.480201 28540 pending_rounds.cc:68] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023063549733756928 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9b8667a49f1a44339122040b0122be0d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } } }
I20240502 01:58:44.480311 28540 raft_consensus.cc:2894] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:58:44.480369 28540 raft_consensus.cc:2866] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:58:44.480420 28540 raft_consensus.cc:2258] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.481786 28540 ts_tablet_manager.cc:1875] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.483633 28540 ts_tablet_manager.cc:1888] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.483712 28540 log.cc:1196] T 9b8667a49f1a44339122040b0122be0d P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b8667a49f1a44339122040b0122be0d
I20240502 01:58:44.484364 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 9b8667a49f1a44339122040b0122be0d (table rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]) successfully deleted
I20240502 01:58:44.489553 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:44.492367 29845 ts_tablet_manager.cc:902] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:44.493120 29845 tablet_copy_client.cc:308] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.7:42021
I20240502 01:58:44.493731 27466 tablet_copy_service.cc:140] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Received BeginTabletCopySession request for tablet a85e6a0806654abd81ca5661a2d019ce from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:51335)
I20240502 01:58:44.493844 27466 tablet_copy_service.cc:161] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Beginning new tablet copy session on tablet a85e6a0806654abd81ca5661a2d019ce from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:51335: session id = f46bcd2bc124471981349b1e2726253b-a85e6a0806654abd81ca5661a2d019ce
I20240502 01:58:44.495036 27466 tablet_copy_source_session.cc:205] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.495136 27557 tablet_service.cc:1467] Processing CreateTablet for tablet f098b9d9b3fc437dbcae3c8529d9c961 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295]), partition=
I20240502 01:58:44.495134 24743 tablet_service.cc:1467] Processing CreateTablet for tablet f098b9d9b3fc437dbcae3c8529d9c961 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295]), partition=
I20240502 01:58:44.495344 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f098b9d9b3fc437dbcae3c8529d9c961. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.495361 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f098b9d9b3fc437dbcae3c8529d9c961. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.497277 29773 tablet_bootstrap.cc:492] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:44.497498 27424 tablet_service.cc:1467] Processing CreateTablet for tablet f098b9d9b3fc437dbcae3c8529d9c961 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295]), partition=
I20240502 01:58:44.497697 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f098b9d9b3fc437dbcae3c8529d9c961. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.497818 29845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85e6a0806654abd81ca5661a2d019ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.498191 29773 tablet_bootstrap.cc:654] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.498212 29853 tablet_bootstrap.cc:492] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:44.499032 29853 tablet_bootstrap.cc:654] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.500471 29774 tablet_bootstrap.cc:492] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:44.500450 29845 tablet_copy_client.cc:783] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.500619 29845 tablet_copy_client.cc:647] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.501412 29774 tablet_bootstrap.cc:654] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.502202 29845 tablet_copy_client.cc:523] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.503474 29845 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:44.506217 29773 tablet_bootstrap.cc:492] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:44.506289 29773 ts_tablet_manager.cc:1367] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:58:44.506680 29773 raft_consensus.cc:348] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.506742 29853 tablet_bootstrap.cc:492] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:44.506783 29773 raft_consensus.cc:374] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.506815 29773 raft_consensus.cc:729] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.506821 29853 ts_tablet_manager.cc:1367] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:44.506953 29773 consensus_queue.cc:260] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.507200 29773 ts_tablet_manager.cc:1398] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:44.509450 29845 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.509655 29845 tablet_bootstrap.cc:492] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:44.509785 29845 ts_tablet_manager.cc:1367] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:58:44.510190 29845 raft_consensus.cc:348] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:44.510208 29853 raft_consensus.cc:348] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.510298 29853 raft_consensus.cc:374] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.510300 29845 raft_consensus.cc:729] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:44.510325 29853 raft_consensus.cc:729] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.510388 29845 consensus_queue.cc:260] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:44.510654 29845 ts_tablet_manager.cc:1398] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.510669 29853 consensus_queue.cc:260] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.511090 29853 ts_tablet_manager.cc:1398] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:44.511874 27466 tablet_copy_service.cc:342] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-a85e6a0806654abd81ca5661a2d019ce received from {username='slave'} at 127.17.243.3:51335
I20240502 01:58:44.511947 27466 tablet_copy_service.cc:433] P 1cc0a6338e0a467dbdc6efb3e67a6a09: ending tablet copy session f46bcd2bc124471981349b1e2726253b-a85e6a0806654abd81ca5661a2d019ce on tablet a85e6a0806654abd81ca5661a2d019ce with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:44.512418 29774 tablet_bootstrap.cc:492] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:44.512498 29774 ts_tablet_manager.cc:1367] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240502 01:58:44.512919 29774 raft_consensus.cc:348] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.513015 29774 raft_consensus.cc:374] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.513044 29774 raft_consensus.cc:729] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.513156 29774 consensus_queue.cc:260] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.513397 29774 ts_tablet_manager.cc:1398] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:44.558461 24483 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 01:58:44.558593 24483 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 01:58:44.561615 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet 693e0959e61c44c59f797e4060187cef
I20240502 01:58:44.561990 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet e148806c0b9143c18ee7938479b481b3
I20240502 01:58:44.562162 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 693e0959e61c44c59f797e4060187cef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:44.562170 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 693e0959e61c44c59f797e4060187cef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.562229 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 4 replicas of tablet 751feb4b2419463c81f3571f6eb0fa42
I20240502 01:58:44.562307 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 693e0959e61c44c59f797e4060187cef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:44.562356 27740 tablet_replica.cc:330] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:44.562464 29434 tablet_replica.cc:330] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:44.562496 27740 raft_consensus.cc:2229] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.562562 29434 raft_consensus.cc:2229] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.562680 29434 raft_consensus.cc:2258] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.562705 27740 raft_consensus.cc:2258] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.562734 28540 tablet_replica.cc:330] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.562876 28540 raft_consensus.cc:2229] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.562880 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 693e0959e61c44c59f797e4060187cef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.563021 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.563043 28540 raft_consensus.cc:2258] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.563138 27864 tablet_replica.cc:330] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.563262 27864 raft_consensus.cc:2229] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.563409 27864 raft_consensus.cc:2258] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.563412 27740 ts_tablet_manager.cc:1875] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.563645 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.564129 27864 ts_tablet_manager.cc:1875] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.564327 29434 ts_tablet_manager.cc:1875] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.565553 27740 ts_tablet_manager.cc:1888] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 01:58:44.565615 27740 log.cc:1196] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/693e0959e61c44c59f797e4060187cef
I20240502 01:58:44.565845 27740 ts_tablet_manager.cc:1909] T 693e0959e61c44c59f797e4060187cef P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:44.566066 27864 ts_tablet_manager.cc:1888] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 01:58:44.566082 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:44.566139 27864 log.cc:1196] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/693e0959e61c44c59f797e4060187cef
I20240502 01:58:44.566380 27864 ts_tablet_manager.cc:1909] T 693e0959e61c44c59f797e4060187cef P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:44.566691 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet e148806c0b9143c18ee7938479b481b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.566761 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet e148806c0b9143c18ee7938479b481b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:52444
W20240502 01:58:44.567106 24473 catalog_manager.cc:4797] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): delete failed for tablet 751feb4b2419463c81f3571f6eb0fa42 because the tablet was not found. No further retry: Not found: Tablet not found: 751feb4b2419463c81f3571f6eb0fa42
I20240502 01:58:44.567119 29434 ts_tablet_manager.cc:1888] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 01:58:44.567180 29434 log.cc:1196] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/693e0959e61c44c59f797e4060187cef
I20240502 01:58:44.567365 29434 ts_tablet_manager.cc:1909] T 693e0959e61c44c59f797e4060187cef P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:44.567504 28027 tablet_replica.cc:330] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.567628 28027 raft_consensus.cc:2229] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.567711 28540 ts_tablet_manager.cc:1875] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.567747 28027 raft_consensus.cc:2258] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.568192 28027 ts_tablet_manager.cc:1875] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.569708 28027 ts_tablet_manager.cc:1888] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:44.569762 28027 log.cc:1196] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/e148806c0b9143c18ee7938479b481b3
I20240502 01:58:44.569752 28540 ts_tablet_manager.cc:1888] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 01:58:44.569809 28540 log.cc:1196] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/693e0959e61c44c59f797e4060187cef
I20240502 01:58:44.569922 28027 ts_tablet_manager.cc:1909] T e148806c0b9143c18ee7938479b481b3 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:44.570176 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 693e0959e61c44c59f797e4060187cef (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.570292 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 693e0959e61c44c59f797e4060187cef (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.570343 25010 tablet_service.cc:1514] Processing DeleteTablet for tablet 751feb4b2419463c81f3571f6eb0fa42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:44.570562 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet e148806c0b9143c18ee7938479b481b3 (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.570789 27740 tablet_replica.cc:330] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:44.570897 27740 raft_consensus.cc:2229] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.571017 27740 raft_consensus.cc:2258] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.571197 28540 ts_tablet_manager.cc:1909] T 693e0959e61c44c59f797e4060187cef P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:44.571420 27894 raft_consensus.cc:1050] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:44.571691 27740 ts_tablet_manager.cc:1875] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.571794 27894 consensus_queue.cc:237] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.571919 28540 tablet_replica.cc:330] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.572041 28540 raft_consensus.cc:2229] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.572168 28540 raft_consensus.cc:2258] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.572360 27864 tablet_replica.cc:330] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.572471 27864 raft_consensus.cc:2229] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.572686 27864 raft_consensus.cc:2258] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.573449 27740 ts_tablet_manager.cc:1888] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:44.573501 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet e148806c0b9143c18ee7938479b481b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:44 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:44.573508 27740 log.cc:1196] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/751feb4b2419463c81f3571f6eb0fa42
I20240502 01:58:44.573709 24763 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.573834 27740 ts_tablet_manager.cc:1909] T 751feb4b2419463c81f3571f6eb0fa42 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:44.574065 27864 ts_tablet_manager.cc:1875] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.574084 27578 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.574096 24897 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.574323 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 693e0959e61c44c59f797e4060187cef (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.574455 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 751feb4b2419463c81f3571f6eb0fa42 (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.574590 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 693e0959e61c44c59f797e4060187cef (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.574640 27740 tablet_replica.cc:330] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:44.574801 27740 raft_consensus.cc:2229] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:44.574975 27740 raft_consensus.cc:2258] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.575351 27740 ts_tablet_manager.cc:1875] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.576421 27864 ts_tablet_manager.cc:1888] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:44.576622 28540 ts_tablet_manager.cc:1875] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.577069 27740 ts_tablet_manager.cc:1888] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:44.577283 27740 log.cc:1196] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/e148806c0b9143c18ee7938479b481b3
I20240502 01:58:44.577543 27740 ts_tablet_manager.cc:1909] T e148806c0b9143c18ee7938479b481b3 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:44.578568 28540 ts_tablet_manager.cc:1888] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:44.578595 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet e148806c0b9143c18ee7938479b481b3 (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.578648 28540 log.cc:1196] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/751feb4b2419463c81f3571f6eb0fa42
I20240502 01:58:44.578938 28540 ts_tablet_manager.cc:1909] T 751feb4b2419463c81f3571f6eb0fa42 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:44.578886 29648 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.579047 27864 log.cc:1196] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/751feb4b2419463c81f3571f6eb0fa42
I20240502 01:58:44.579267 29648 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.579340 27864 ts_tablet_manager.cc:1909] T 751feb4b2419463c81f3571f6eb0fa42 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:44.580014 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 751feb4b2419463c81f3571f6eb0fa42 (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.580437 28540 tablet_replica.cc:330] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.580552 28540 raft_consensus.cc:2229] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.580688 28540 raft_consensus.cc:2258] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.580708 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 751feb4b2419463c81f3571f6eb0fa42 (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.580683 27890 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.581159 28540 ts_tablet_manager.cc:1875] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:44.582439 29648 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.582955 28540 ts_tablet_manager.cc:1888] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:58:44.583030 28540 log.cc:1196] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/e148806c0b9143c18ee7938479b481b3
I20240502 01:58:44.583614 28540 ts_tablet_manager.cc:1909] T e148806c0b9143c18ee7938479b481b3 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:44.584161 24763 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.584306 24897 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.585522 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet e148806c0b9143c18ee7938479b481b3 (table rebalance_test_table_18 [id=baa6fe8d075146fcae47c0b55b938e5e]) successfully deleted
I20240502 01:58:44.586261 24483 catalog_manager.cc:5484] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.590407 29916 ts_tablet_manager.cc:902] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:44.591840 27576 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.592625 29916 tablet_copy_client.cc:308] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:44.593101 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 488816b0d5d542bcb5fa12ff7e70fd40 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:54405)
I20240502 01:58:44.593183 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 488816b0d5d542bcb5fa12ff7e70fd40 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:54405: session id = 9dd619a595e94004ac0f7f5a9054e7cc-488816b0d5d542bcb5fa12ff7e70fd40
I20240502 01:58:44.594198 25054 tablet_copy_source_session.cc:205] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.595042 29916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488816b0d5d542bcb5fa12ff7e70fd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.597671 25032 consensus_queue.cc:237] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.598822 24763 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.598824 27576 raft_consensus.cc:1261] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.598940 29916 tablet_copy_client.cc:783] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.599092 29916 tablet_copy_client.cc:647] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.599412 27894 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.599980 28534 consensus_queue.cc:1035] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.600809 29916 tablet_copy_client.cc:523] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.601430 27894 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.601821 27576 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.602088 29916 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:44.603111 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 086c9b674a0543cab3c82349d151e373 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.603912 24478 catalog_manager.cc:5484] T 086c9b674a0543cab3c82349d151e373 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.604808 24763 raft_consensus.cc:2932] T 086c9b674a0543cab3c82349d151e373 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.607189 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 086c9b674a0543cab3c82349d151e373 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:44.607367 28027 tablet_replica.cc:330] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:44.607483 28027 raft_consensus.cc:2229] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.607607 28027 raft_consensus.cc:2258] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.609316 28027 ts_tablet_manager.cc:1875] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.611310 28027 ts_tablet_manager.cc:1888] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.611367 28027 log.cc:1196] T 086c9b674a0543cab3c82349d151e373 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/086c9b674a0543cab3c82349d151e373
I20240502 01:58:44.611850 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 086c9b674a0543cab3c82349d151e373 (table rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]) successfully deleted
I20240502 01:58:44.618017 29658 raft_consensus.cc:1050] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:44.618430 29658 consensus_queue.cc:237] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.621125 27446 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.621613 29929 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.621754 28568 raft_consensus.cc:482] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:44.621845 28568 raft_consensus.cc:504] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.623812 24763 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.623956 24897 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.624008 28568 leader_election.cc:288] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:44.624210 29929 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.624413 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f098b9d9b3fc437dbcae3c8529d9c961" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:44.624547 27576 raft_consensus.cc:2454] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:44.624703 29929 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.624832 24701 leader_election.cc:302] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:44.625514 28568 raft_consensus.cc:2781] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:44.625572 28568 raft_consensus.cc:482] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:44.625612 28568 raft_consensus.cc:3037] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.626109 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f098b9d9b3fc437dbcae3c8529d9c961" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:44.626220 27446 raft_consensus.cc:2454] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:44.626547 28568 raft_consensus.cc:504] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.626868 28568 leader_election.cc:288] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:44.627470 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f098b9d9b3fc437dbcae3c8529d9c961" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:44.627557 27576 raft_consensus.cc:3037] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.627821 28799 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.628418 27576 raft_consensus.cc:2454] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:44.628333 24897 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.628911 24701 leader_election.cc:302] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:44.629202 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f098b9d9b3fc437dbcae3c8529d9c961" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:44.629298 27446 raft_consensus.cc:3037] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:44.630189 27446 raft_consensus.cc:2454] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:44.630750 28568 raft_consensus.cc:2781] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:44.630844 28568 raft_consensus.cc:686] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:44.630954 28568 consensus_queue.cc:237] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:44.631186 24479 catalog_manager.cc:5484] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.633119 24478 catalog_manager.cc:5484] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.633118 27446 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.636229 24630 consensus_queue.cc:237] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.637261 27446 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.638484 28396 raft_consensus.cc:1050] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:44.638474 29929 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.638866 28396 consensus_queue.cc:237] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.638924 27446 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:44.640221 24763 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.640669 29573 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.640861 28489 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.641140 27445 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.641247 27311 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.641660 29916 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.641649 29573 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.641954 29916 tablet_bootstrap.cc:492] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:44.642125 29916 ts_tablet_manager.cc:1367] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.000s
I20240502 01:58:44.642527 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 645305c993a5486ebc1e6c8b81941128 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.642530 29916 raft_consensus.cc:348] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.642640 29916 raft_consensus.cc:729] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:44.642725 29916 consensus_queue.cc:260] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.643008 29916 ts_tablet_manager.cc:1398] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:44.643518 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-488816b0d5d542bcb5fa12ff7e70fd40 received from {username='slave'} at 127.17.243.5:54405
I20240502 01:58:44.643587 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-488816b0d5d542bcb5fa12ff7e70fd40 on tablet 488816b0d5d542bcb5fa12ff7e70fd40 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:44.644019 24630 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.644352 24483 catalog_manager.cc:5484] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.644469 29573 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:58:44.646283 27377 consensus_peers.cc:485] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): Couldn't send request to peer e0b9e7df4e8543fa9b8e3ffc24330657. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adc992deb712458aa793eaa11019de1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.646535 27312 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.647024 28447 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.647383 27576 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.647733 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 645305c993a5486ebc1e6c8b81941128 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.647891 29244 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.648033 27864 tablet_replica.cc:330] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.648154 27864 raft_consensus.cc:2229] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.648262 27864 pending_rounds.cc:62] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Trying to abort 1 pending ops.
I20240502 01:58:44.648301 27864 pending_rounds.cc:68] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023063550435987456 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "645305c993a5486ebc1e6c8b81941128" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } } }
I20240502 01:58:44.648427 27864 raft_consensus.cc:2894] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:58:44.648484 27864 raft_consensus.cc:2866] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:58:44.648526 27864 raft_consensus.cc:2258] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.648785 28447 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:44.649351 27864 ts_tablet_manager.cc:1875] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.649624 24897 raft_consensus.cc:1261] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.650524 27312 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:44.651654 27864 ts_tablet_manager.cc:1888] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.651713 27864 log.cc:1196] T 645305c993a5486ebc1e6c8b81941128 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/645305c993a5486ebc1e6c8b81941128
I20240502 01:58:44.652783 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 645305c993a5486ebc1e6c8b81941128 (table rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]) successfully deleted
I20240502 01:58:44.653021 24478 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:44.654191 29573 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.654296 29929 consensus_queue.cc:1035] T 645305c993a5486ebc1e6c8b81941128 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.655222 24897 raft_consensus.cc:2932] T 645305c993a5486ebc1e6c8b81941128 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.656252 24630 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.658000 27311 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.658041 24478 catalog_manager.cc:5484] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.659981 24763 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.663133 27576 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:44.664220 24897 consensus_queue.cc:237] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.665367 27311 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.665503 24763 raft_consensus.cc:1261] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.665712 29574 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.666060 29574 consensus_queue.cc:1035] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.667380 29573 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.667929 24763 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.668715 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16ae91433f10465c922183298a387caa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.668897 27311 raft_consensus.cc:2932] T 16ae91433f10465c922183298a387caa P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.670481 24483 catalog_manager.cc:5484] T 16ae91433f10465c922183298a387caa P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 5 to 6, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.672729 24897 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:44.673425 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ae91433f10465c922183298a387caa with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:44.673615 28540 tablet_replica.cc:330] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:44.673734 28540 raft_consensus.cc:2229] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.673884 28540 raft_consensus.cc:2258] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.674171 27446 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:44.674206 24630 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:58:44.674612 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3edec06b3d8450a8cc3002be70c848a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.674702 28540 ts_tablet_manager.cc:1875] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.674710 29579 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:44.675125 29579 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:44.676429 28540 ts_tablet_manager.cc:1888] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.676499 28540 log.cc:1196] T 16ae91433f10465c922183298a387caa P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/16ae91433f10465c922183298a387caa
I20240502 01:58:44.676782 29574 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:44.676968 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 16ae91433f10465c922183298a387caa (table rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]) successfully deleted
I20240502 01:58:44.677170 24630 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:44.679231 27446 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:44.679430 24483 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.687270 24897 consensus_queue.cc:237] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.688373 24763 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.688524 25032 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:44.688612 24832 consensus_peers.cc:485] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d136e006dd1048739156526af42fddd1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.688779 29578 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.688939 29574 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.690281 29579 raft_consensus.cc:1050] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:44.690621 29579 consensus_queue.cc:237] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.690824 27093 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.692090 25032 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.691987 24763 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.692157 24765 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.692478 27446 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.692545 29579 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.692829 29579 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.692734 25032 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.694037 29574 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.695251 24478 catalog_manager.cc:5484] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:44.697991 28396 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.698565 27446 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.700147 25032 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.700489 24763 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.700997 24483 catalog_manager.cc:5484] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.704336 24897 consensus_queue.cc:237] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:44.705339 25032 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.705422 27446 raft_consensus.cc:1261] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:44.705658 29579 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.705808 28396 consensus_queue.cc:1035] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:44.707756 29579 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.709203 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d5f6bcbe0a048eb85b9475348b155ef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.709336 25032 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.709868 24478 catalog_manager.cc:5484] T 8d5f6bcbe0a048eb85b9475348b155ef P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.711109 27446 raft_consensus.cc:2932] T 8d5f6bcbe0a048eb85b9475348b155ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:44.712256 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5f6bcbe0a048eb85b9475348b155ef with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.712448 27864 tablet_replica.cc:330] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.712568 27864 raft_consensus.cc:2229] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.712706 27864 raft_consensus.cc:2258] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.713598 27864 ts_tablet_manager.cc:1875] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.715368 27864 ts_tablet_manager.cc:1888] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:44.715444 27864 log.cc:1196] T 8d5f6bcbe0a048eb85b9475348b155ef P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d5f6bcbe0a048eb85b9475348b155ef
I20240502 01:58:44.715878 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 8d5f6bcbe0a048eb85b9475348b155ef (table rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]) successfully deleted
I20240502 01:58:44.745148 29929 raft_consensus.cc:1050] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:44.745563 29929 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.746987 27177 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:44.747560 27576 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:44.747537 29658 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:44.747651 24763 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:44.748185 29658 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:44.748572 28799 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:44.750326 29658 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.752840 24478 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.752934 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:44.752920 24763 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.755450 27576 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.755736 27177 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.758303 27446 raft_consensus.cc:1261] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.758589 24630 consensus_queue.cc:237] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:44.758646 28568 consensus_queue.cc:1035] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.760330 27578 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:44.761394 28489 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:44.761826 27576 raft_consensus.cc:1261] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.762395 27177 raft_consensus.cc:1261] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:44.762660 28393 consensus_queue.cc:1035] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.763003 29658 consensus_queue.cc:1035] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:44.763249 29610 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:44.763381 24479 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet f098b9d9b3fc437dbcae3c8529d9c961 (table rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295]). Expected version 1 got 0
I20240502 01:58:44.763543 29610 tablet.cc:1620] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:44.764393 29247 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:44.764652 29247 tablet.cc:1620] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:44.766371 29845 ts_tablet_manager.cc:902] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:44.766374 29658 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.766867 29845 tablet_copy_client.cc:308] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.7:42021
I20240502 01:58:44.767441 27466 tablet_copy_service.cc:140] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Received BeginTabletCopySession request for tablet b6c09e8f421d4ae2adf7c9b63121943c from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:51335)
I20240502 01:58:44.767524 27466 tablet_copy_service.cc:161] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Beginning new tablet copy session on tablet b6c09e8f421d4ae2adf7c9b63121943c from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:51335: session id = f46bcd2bc124471981349b1e2726253b-b6c09e8f421d4ae2adf7c9b63121943c
I20240502 01:58:44.767789 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e720d8363bb049938798ab0b9aabc28d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:44.768399 27466 tablet_copy_source_session.cc:205] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.769368 24478 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet f098b9d9b3fc437dbcae3c8529d9c961 (table rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295]). Expected version 1 got 0
I20240502 01:58:44.770051 24483 catalog_manager.cc:5484] T e720d8363bb049938798ab0b9aabc28d P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:44.770507 29845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c09e8f421d4ae2adf7c9b63121943c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.771215 27177 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.771703 27578 raft_consensus.cc:2932] T e720d8363bb049938798ab0b9aabc28d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:44.772707 29845 tablet_copy_client.cc:783] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.772856 29845 tablet_copy_client.cc:647] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.773425 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295] alter complete (version 1)
I20240502 01:58:44.774000 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet e720d8363bb049938798ab0b9aabc28d with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:44.774606 27864 tablet_replica.cc:330] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:44.774739 27864 raft_consensus.cc:2229] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:44.774878 27864 raft_consensus.cc:2258] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:44.775542 27864 ts_tablet_manager.cc:1875] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:44.777303 27864 ts_tablet_manager.cc:1888] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:58:44.777352 27864 log.cc:1196] T e720d8363bb049938798ab0b9aabc28d P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/e720d8363bb049938798ab0b9aabc28d
I20240502 01:58:44.778183 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet e720d8363bb049938798ab0b9aabc28d (table rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]) successfully deleted
I20240502 01:58:44.778956 29845 tablet_copy_client.cc:523] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.780185 29845 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:44.783139 24478 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet f098b9d9b3fc437dbcae3c8529d9c961 (table rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295]). Expected version 1 got 0
I20240502 01:58:44.784034 30028 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:44.784289 29845 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.784298 30028 tablet.cc:1620] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:44.785205 29845 tablet_bootstrap.cc:492] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:44.785355 29845 ts_tablet_manager.cc:1367] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:44.785564 29916 ts_tablet_manager.cc:902] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:44.785775 29845 raft_consensus.cc:348] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:44.785882 29845 raft_consensus.cc:729] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:44.786031 29916 tablet_copy_client.cc:308] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:44.785995 29845 consensus_queue.cc:260] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:44.786274 29845 ts_tablet_manager.cc:1398] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.786727 27466 tablet_copy_service.cc:342] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-b6c09e8f421d4ae2adf7c9b63121943c received from {username='slave'} at 127.17.243.3:51335
I20240502 01:58:44.786793 27466 tablet_copy_service.cc:433] P 1cc0a6338e0a467dbdc6efb3e67a6a09: ending tablet copy session f46bcd2bc124471981349b1e2726253b-b6c09e8f421d4ae2adf7c9b63121943c on tablet b6c09e8f421d4ae2adf7c9b63121943c with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:44.787137 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 26b7501639064035a570b3b4c1cf083f from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:54405)
I20240502 01:58:44.787226 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 26b7501639064035a570b3b4c1cf083f from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:54405: session id = 9dd619a595e94004ac0f7f5a9054e7cc-26b7501639064035a570b3b4c1cf083f
I20240502 01:58:44.788093 25054 tablet_copy_source_session.cc:205] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.789486 29916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7501639064035a570b3b4c1cf083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.791241 29916 tablet_copy_client.cc:783] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.791920 29916 tablet_copy_client.cc:647] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.793110 29916 tablet_copy_client.cc:523] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.794781 29916 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:44.818435 29916 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.818671 29916 tablet_bootstrap.cc:492] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:44.818840 29916 ts_tablet_manager.cc:1367] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.001s
I20240502 01:58:44.819267 29916 raft_consensus.cc:348] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.819380 29916 raft_consensus.cc:729] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:44.819490 29916 consensus_queue.cc:260] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:44.819772 29916 ts_tablet_manager.cc:1398] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:44.820801 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-26b7501639064035a570b3b4c1cf083f received from {username='slave'} at 127.17.243.5:54405
I20240502 01:58:44.820879 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-26b7501639064035a570b3b4c1cf083f on tablet 26b7501639064035a570b3b4c1cf083f with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:44.826503 27311 consensus_queue.cc:237] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:44.827719 29658 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.828269 27576 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.828648 27446 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.829424 30063 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.829756 29605 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.830211 28315 consensus_queue.cc:1035] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.832101 29929 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:44.833141 27246 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f -> Peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Couldn't send request to peer 92cb785c165749eca49af5784756461c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2078a814ae1648e2bde5e521ef45af6d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.834540 29605 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:44.837736 27578 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:44.838115 27446 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:44.838236 28447 consensus_queue.cc:1035] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.839458 24479 catalog_manager.cc:5484] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:44.848578 24763 consensus_queue.cc:237] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } }
I20240502 01:58:44.849846 27311 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:44.849949 27446 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:44.850196 24700 consensus_peers.cc:485] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c -> Peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): Couldn't send request to peer 588bfc090a5247598cd5d3cee5ad2007. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6186e8ffddc47a4a3b428fcd7ce19fb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.850270 28393 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.850452 30082 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:44.852035 28568 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:44.852488 27446 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:44.852610 27311 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:44.854785 24478 catalog_manager.cc:5484] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:44.861393 24897 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:44.862627 24763 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:44.862728 24632 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:44.863178 24833 consensus_peers.cc:485] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93695e2165e142f79293ac14e4f4fb84. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.863380 29578 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.863890 29578 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:44.865037 29579 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.865373 24763 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.866089 24632 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.868686 24486 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:44.870561 27890 consensus_queue.cc:1035] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.874020 27894 consensus_queue.cc:1035] T dad6070e3df4409a9e684a435e780799 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.879912 24478 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:44.884192 29610 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:44.884528 30028 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:44.884797 30028 tablet.cc:1620] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:44.884500 29610 tablet.cc:1620] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:44.885417 29247 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:44.885680 29247 tablet.cc:1620] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:44.888761 30107 ts_tablet_manager.cc:902] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:44.889232 30107 tablet_copy_client.cc:308] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:44.889842 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9d2fc9e8407a4dff8a780a1845a0d295] alter complete (version 2)
I20240502 01:58:44.897644 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 44be529dec974eac8419efe9c19b30bf from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:41115)
I20240502 01:58:44.897743 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 44be529dec974eac8419efe9c19b30bf from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:41115: session id = bb653087e8554c7591cf547dd969f3be-44be529dec974eac8419efe9c19b30bf
I20240502 01:58:44.898782 24918 tablet_copy_source_session.cc:205] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.902547 30107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44be529dec974eac8419efe9c19b30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.904620 30107 tablet_copy_client.cc:783] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.904781 30107 tablet_copy_client.cc:647] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.906948 30107 tablet_copy_client.cc:523] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.908149 30107 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:44.929046 30107 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.929337 30107 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:44.929589 30107 ts_tablet_manager.cc:1367] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.005s
I20240502 01:58:44.930042 30107 raft_consensus.cc:348] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:44.930171 30107 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:44.930303 30107 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:44.930640 30107 ts_tablet_manager.cc:1398] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.931427 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-44be529dec974eac8419efe9c19b30bf received from {username='slave'} at 127.17.243.8:41115
I20240502 01:58:44.931504 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session bb653087e8554c7591cf547dd969f3be-44be529dec974eac8419efe9c19b30bf on tablet 44be529dec974eac8419efe9c19b30bf with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:44.954618 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:44.961443 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2686f7b58341e9a0b1f2b27cfbb173 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.961558 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2686f7b58341e9a0b1f2b27cfbb173 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.961786 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2686f7b58341e9a0b1f2b27cfbb173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.961845 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2686f7b58341e9a0b1f2b27cfbb173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.962015 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2686f7b58341e9a0b1f2b27cfbb173 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.962275 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2686f7b58341e9a0b1f2b27cfbb173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.963711 30145 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:44.964215 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 8919df9a8a574882bad4995fc5df7e50 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.964470 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8919df9a8a574882bad4995fc5df7e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.964670 30145 tablet_bootstrap.cc:654] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.964690 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 8919df9a8a574882bad4995fc5df7e50 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.964995 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8919df9a8a574882bad4995fc5df7e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.966074 29774 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:44.966478 29773 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:44.966846 29774 tablet_bootstrap.cc:654] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.967376 29773 tablet_bootstrap.cc:654] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.967473 30147 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:44.967567 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 356b84e895db449d990b1e3c45d60159 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.967829 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356b84e895db449d990b1e3c45d60159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.968041 29774 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:44.968125 29774 ts_tablet_manager.cc:1367] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:44.968530 29774 raft_consensus.cc:348] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.968632 29774 raft_consensus.cc:374] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.968626 30147 tablet_bootstrap.cc:654] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.968662 29774 raft_consensus.cc:729] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.968770 29774 consensus_queue.cc:260] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.969046 29774 ts_tablet_manager.cc:1398] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:44.969133 29773 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:44.969201 29774 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:44.969220 29773 ts_tablet_manager.cc:1367] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:44.969631 29773 raft_consensus.cc:348] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.969709 29773 raft_consensus.cc:374] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.969732 29773 raft_consensus.cc:729] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.969841 29773 consensus_queue.cc:260] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.970046 30151 tablet_bootstrap.cc:492] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:44.970176 29773 ts_tablet_manager.cc:1398] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:44.970225 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 356b84e895db449d990b1e3c45d60159 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.970291 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 8919df9a8a574882bad4995fc5df7e50 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.970480 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8919df9a8a574882bad4995fc5df7e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.970489 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356b84e895db449d990b1e3c45d60159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.971007 30151 tablet_bootstrap.cc:654] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.971907 30147 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:44.971974 30147 ts_tablet_manager.cc:1367] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:44.972353 30147 raft_consensus.cc:348] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:44.972530 30147 raft_consensus.cc:374] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.972571 30147 raft_consensus.cc:729] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.972684 30153 tablet_bootstrap.cc:492] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:44.972697 30147 consensus_queue.cc:260] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:44.972868 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 356b84e895db449d990b1e3c45d60159 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=6f9c826b8d0e402ca78dc0842fae2505]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:44.973073 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356b84e895db449d990b1e3c45d60159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.973079 30147 ts_tablet_manager.cc:1398] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.973793 30153 tablet_bootstrap.cc:654] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.975921 30151 tablet_bootstrap.cc:492] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:44.976023 30151 ts_tablet_manager.cc:1367] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:44.976351 30153 tablet_bootstrap.cc:492] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:44.976429 30153 ts_tablet_manager.cc:1367] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:44.976473 30151 raft_consensus.cc:348] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.976576 30151 raft_consensus.cc:374] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.976608 30151 raft_consensus.cc:729] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.976734 30151 consensus_queue.cc:260] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.976915 30145 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:44.976913 30153 raft_consensus.cc:348] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.976994 30145 ts_tablet_manager.cc:1367] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240502 01:58:44.977010 30153 raft_consensus.cc:374] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.977022 30151 ts_tablet_manager.cc:1398] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.977043 30153 raft_consensus.cc:729] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.977169 30153 consensus_queue.cc:260] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.977458 30153 ts_tablet_manager.cc:1398] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.977442 30145 raft_consensus.cc:348] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.977540 30145 raft_consensus.cc:374] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.977571 30145 raft_consensus.cc:729] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.977689 30145 consensus_queue.cc:260] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:44.977995 30145 ts_tablet_manager.cc:1398] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:44.978174 30145 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:44.978170 29774 tablet_bootstrap.cc:654] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.978986 30145 tablet_bootstrap.cc:654] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.980060 30145 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:44.980208 30145 ts_tablet_manager.cc:1367] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:44.980942 30145 raft_consensus.cc:348] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:44.981539 30145 raft_consensus.cc:374] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.981631 30145 raft_consensus.cc:729] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.981887 30145 consensus_queue.cc:260] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:44.982430 30145 ts_tablet_manager.cc:1398] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:44.982651 30145 tablet_bootstrap.cc:492] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:44.984151 29774 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:44.984233 29774 ts_tablet_manager.cc:1367] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240502 01:58:44.984431 30145 tablet_bootstrap.cc:654] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:44.985908 30145 tablet_bootstrap.cc:492] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:44.986016 30145 ts_tablet_manager.cc:1367] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:44.986471 30145 raft_consensus.cc:348] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.986593 30145 raft_consensus.cc:374] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.986629 30145 raft_consensus.cc:729] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.986729 30145 consensus_queue.cc:260] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:44.984644 29774 raft_consensus.cc:348] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:44.986876 29774 raft_consensus.cc:374] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:44.986927 29774 raft_consensus.cc:729] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.986992 30145 ts_tablet_manager.cc:1398] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:44.987047 29774 consensus_queue.cc:260] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:44.987282 29774 ts_tablet_manager.cc:1398] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:44.988693 29579 raft_consensus.cc:1050] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:44.989113 29579 consensus_queue.cc:237] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:44.990518 25034 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.990749 24763 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.991055 28396 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.991354 27093 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.991725 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:44.992918 27311 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:44.993820 27093 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:44.996517 27093 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.997828 27311 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:44.998709 29610 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:44.999394 29247 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:44.999840 24478 catalog_manager.cc:5484] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.000950 25034 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:45.001871 30028 tablet.cc:1722] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:45.004473 24763 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:45.006130 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=9d2fc9e8407a4dff8a780a1845a0d295] alter complete (version 3)
I20240502 01:58:45.007977 24897 consensus_queue.cc:237] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:45.009506 27311 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:45.009507 25034 raft_consensus.cc:1261] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:45.009970 29574 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:45.010180 29579 consensus_queue.cc:1035] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:45.012192 29578 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:45.012728 25034 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:45.014123 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48085cebf94e4f549e6174a88acdcc59 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:45.014736 27311 raft_consensus.cc:2932] T 48085cebf94e4f549e6174a88acdcc59 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:45.015590 24483 catalog_manager.cc:5484] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.018723 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 48085cebf94e4f549e6174a88acdcc59 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:45.019199 27864 tablet_replica.cc:330] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:45.019358 27864 raft_consensus.cc:2229] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.019526 27864 raft_consensus.cc:2258] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.020145 27864 ts_tablet_manager.cc:1875] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:45.022243 27864 ts_tablet_manager.cc:1888] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:45.022315 27864 log.cc:1196] T 48085cebf94e4f549e6174a88acdcc59 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/48085cebf94e4f549e6174a88acdcc59
I20240502 01:58:45.022763 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 48085cebf94e4f549e6174a88acdcc59 (table rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]) successfully deleted
I20240502 01:58:45.022836 30184 ts_tablet_manager.cc:902] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:45.023331 30184 tablet_copy_client.cc:308] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:45.023736 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 89580549c13d4c1ebdb2a5d24da2a01e from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:55633)
I20240502 01:58:45.023826 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 89580549c13d4c1ebdb2a5d24da2a01e from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:55633: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-89580549c13d4c1ebdb2a5d24da2a01e
I20240502 01:58:45.024740 24918 tablet_copy_source_session.cc:205] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.025558 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89580549c13d4c1ebdb2a5d24da2a01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.027936 30184 tablet_copy_client.cc:783] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.028091 30184 tablet_copy_client.cc:647] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.029137 30184 tablet_copy_client.cc:523] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.030274 30184 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:45.051702 30184 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.051957 30184 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:45.052109 30184 ts_tablet_manager.cc:1367] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20240502 01:58:45.052457 30184 raft_consensus.cc:348] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:45.052557 30184 raft_consensus.cc:729] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:45.052650 30184 consensus_queue.cc:260] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:45.052904 30184 ts_tablet_manager.cc:1398] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:45.053407 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-89580549c13d4c1ebdb2a5d24da2a01e received from {username='slave'} at 127.17.243.7:55633
I20240502 01:58:45.053505 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-89580549c13d4c1ebdb2a5d24da2a01e on tablet 89580549c13d4c1ebdb2a5d24da2a01e with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:45.066704 24897 raft_consensus.cc:1203] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:45.074110 30195 ts_tablet_manager.cc:902] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:45.074604 30195 tablet_copy_client.cc:308] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:45.075070 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet b3edec06b3d8450a8cc3002be70c848a from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:44697)
I20240502 01:58:45.075168 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet b3edec06b3d8450a8cc3002be70c848a from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:44697: session id = 1049b4c8783b48f39be22bd113ad349f-b3edec06b3d8450a8cc3002be70c848a
I20240502 01:58:45.075893 24918 tablet_copy_source_session.cc:205] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.076709 30195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3edec06b3d8450a8cc3002be70c848a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.078657 30195 tablet_copy_client.cc:783] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.078814 30195 tablet_copy_client.cc:647] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.079891 30195 tablet_copy_client.cc:523] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.080935 30195 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:45.081528 29244 raft_consensus.cc:482] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.081632 29244 raft_consensus.cc:504] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.082098 29244 leader_election.cc:288] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:45.082352 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2686f7b58341e9a0b1f2b27cfbb173" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:45.082471 27311 raft_consensus.cc:2454] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:45.082923 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2686f7b58341e9a0b1f2b27cfbb173" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:45.083042 27377 leader_election.cc:302] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:45.083078 27576 raft_consensus.cc:2454] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:45.083472 29244 raft_consensus.cc:2781] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:45.083532 29244 raft_consensus.cc:482] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:45.083559 29244 raft_consensus.cc:3037] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.084354 29244 raft_consensus.cc:504] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.084668 29244 leader_election.cc:288] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:45.084973 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2686f7b58341e9a0b1f2b27cfbb173" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:45.085054 27311 raft_consensus.cc:3037] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.085191 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2686f7b58341e9a0b1f2b27cfbb173" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:45.085301 27576 raft_consensus.cc:3037] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.085635 27311 raft_consensus.cc:2454] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:45.085912 27377 leader_election.cc:302] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:45.086086 29244 raft_consensus.cc:2781] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:45.086161 29244 raft_consensus.cc:686] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:45.086279 27576 raft_consensus.cc:2454] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:45.086284 29244 consensus_queue.cc:237] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.088002 24486 catalog_manager.cc:5484] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:45.095530 27177 raft_consensus.cc:1203] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:45.101521 30195 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.101749 30195 tablet_bootstrap.cc:492] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:45.101898 30195 ts_tablet_manager.cc:1367] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20240502 01:58:45.102288 30195 raft_consensus.cc:348] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:45.102384 30195 raft_consensus.cc:729] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:45.102470 30195 consensus_queue.cc:260] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:45.102717 30195 ts_tablet_manager.cc:1398] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:45.103160 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-b3edec06b3d8450a8cc3002be70c848a received from {username='slave'} at 127.17.243.6:44697
I20240502 01:58:45.103233 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-b3edec06b3d8450a8cc3002be70c848a on tablet b3edec06b3d8450a8cc3002be70c848a with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:45.109742 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:45.109920 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:45.111960 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet f098b9d9b3fc437dbcae3c8529d9c961
I20240502 01:58:45.112428 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet f098b9d9b3fc437dbcae3c8529d9c961 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:45 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:45.112939 30202 tablet_replica.cc:330] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:45.112936 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet f098b9d9b3fc437dbcae3c8529d9c961 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:45 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:45.113101 30202 raft_consensus.cc:2229] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.113162 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet f098b9d9b3fc437dbcae3c8529d9c961 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:45 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:45.113225 30202 raft_consensus.cc:2258] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.113343 27864 tablet_replica.cc:330] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:45.113472 27864 raft_consensus.cc:2229] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:45.113677 27864 raft_consensus.cc:2258] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.113827 30202 ts_tablet_manager.cc:1875] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:45.114276 27864 ts_tablet_manager.cc:1875] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:45.115002 30203 tablet_replica.cc:330] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:45.115130 30203 raft_consensus.cc:2229] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.115271 30203 raft_consensus.cc:2258] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.116045 30202 ts_tablet_manager.cc:1888] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:45.116109 30202 log.cc:1196] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/f098b9d9b3fc437dbcae3c8529d9c961
I20240502 01:58:45.116329 27864 ts_tablet_manager.cc:1888] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:45.116348 30202 ts_tablet_manager.cc:1909] T f098b9d9b3fc437dbcae3c8529d9c961 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:45.116382 27864 log.cc:1196] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/f098b9d9b3fc437dbcae3c8529d9c961
I20240502 01:58:45.116556 27864 ts_tablet_manager.cc:1909] T f098b9d9b3fc437dbcae3c8529d9c961 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:45.116923 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet f098b9d9b3fc437dbcae3c8529d9c961 (table rebalancer_dynamic_table_new_name [id=9d2fc9e8407a4dff8a780a1845a0d295]) successfully deleted
I20240502 01:58:45.117322 30203 ts_tablet_manager.cc:1875] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:45.117544 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet f098b9d9b3fc437dbcae3c8529d9c961 (table rebalancer_dynamic_table_new_name [id=9d2fc9e8407a4dff8a780a1845a0d295]) successfully deleted
I20240502 01:58:45.119174 30203 ts_tablet_manager.cc:1888] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:45.119240 30203 log.cc:1196] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/f098b9d9b3fc437dbcae3c8529d9c961
I20240502 01:58:45.119462 30203 ts_tablet_manager.cc:1909] T f098b9d9b3fc437dbcae3c8529d9c961 P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:45.120021 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet f098b9d9b3fc437dbcae3c8529d9c961 (table rebalancer_dynamic_table_new_name [id=9d2fc9e8407a4dff8a780a1845a0d295]) successfully deleted
I20240502 01:58:45.122020 30204 ts_tablet_manager.cc:902] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: Initiating tablet copy from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:45.122557 30204 tablet_copy_client.cc:308] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.7:42021
I20240502 01:58:45.123188 27466 tablet_copy_service.cc:140] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Received BeginTabletCopySession request for tablet adc992deb712458aa793eaa11019de1f from peer e0b9e7df4e8543fa9b8e3ffc24330657 ({username='slave'} at 127.17.243.4:45579)
I20240502 01:58:45.123265 27466 tablet_copy_service.cc:161] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Beginning new tablet copy session on tablet adc992deb712458aa793eaa11019de1f from peer e0b9e7df4e8543fa9b8e3ffc24330657 at {username='slave'} at 127.17.243.4:45579: session id = e0b9e7df4e8543fa9b8e3ffc24330657-adc992deb712458aa793eaa11019de1f
I20240502 01:58:45.123869 27466 tablet_copy_source_session.cc:205] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.124753 30204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc992deb712458aa793eaa11019de1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.126765 30204 tablet_copy_client.cc:783] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.126920 30204 tablet_copy_client.cc:647] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.128402 30204 tablet_copy_client.cc:523] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.129637 30204 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:45.132627 30204 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.132820 30204 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap complete.
I20240502 01:58:45.132952 30204 ts_tablet_manager.cc:1367] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:45.133316 30204 raft_consensus.cc:348] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:45.133414 30204 raft_consensus.cc:729] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: LEARNER
I20240502 01:58:45.133524 30204 consensus_queue.cc:260] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:45.133793 30204 ts_tablet_manager.cc:1398] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 01:58:45.134305 27466 tablet_copy_service.cc:342] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Request end of tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-adc992deb712458aa793eaa11019de1f received from {username='slave'} at 127.17.243.4:45579
I20240502 01:58:45.134369 27466 tablet_copy_service.cc:433] P 1cc0a6338e0a467dbdc6efb3e67a6a09: ending tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-adc992deb712458aa793eaa11019de1f on tablet adc992deb712458aa793eaa11019de1f with peer e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:58:45.194048 28799 raft_consensus.cc:482] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.194185 28799 raft_consensus.cc:504] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:45.194568 28799 leader_election.cc:288] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:45.194960 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356b84e895db449d990b1e3c45d60159" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:45.194993 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356b84e895db449d990b1e3c45d60159" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:45.195099 27311 raft_consensus.cc:2454] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:45.195128 25034 raft_consensus.cc:2454] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:45.195425 24564 leader_election.cc:302] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:45.195632 28799 raft_consensus.cc:2781] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:45.195691 28799 raft_consensus.cc:482] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:45.195734 28799 raft_consensus.cc:3037] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.196511 28799 raft_consensus.cc:504] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:45.196811 28799 leader_election.cc:288] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:45.197132 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356b84e895db449d990b1e3c45d60159" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:45.197153 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356b84e895db449d990b1e3c45d60159" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:45.197233 27311 raft_consensus.cc:3037] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.197238 25034 raft_consensus.cc:3037] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.197921 25034 raft_consensus.cc:2454] T 356b84e895db449d990b1e3c45d60159 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:45.198016 27311 raft_consensus.cc:2454] T 356b84e895db449d990b1e3c45d60159 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:45.198231 24564 leader_election.cc:302] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:45.198494 28799 raft_consensus.cc:2781] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:45.198590 28799 raft_consensus.cc:686] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:45.198719 28799 consensus_queue.cc:237] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:45.200436 24486 catalog_manager.cc:5484] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.205438 30211 ts_tablet_manager.cc:902] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: Initiating tablet copy from peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:45.206183 30211 tablet_copy_client.cc:308] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.6:41707
I20240502 01:58:45.206799 27332 tablet_copy_service.cc:140] P 1049b4c8783b48f39be22bd113ad349f: Received BeginTabletCopySession request for tablet 2078a814ae1648e2bde5e521ef45af6d from peer 92cb785c165749eca49af5784756461c ({username='slave'} at 127.17.243.2:42935)
I20240502 01:58:45.206923 27332 tablet_copy_service.cc:161] P 1049b4c8783b48f39be22bd113ad349f: Beginning new tablet copy session on tablet 2078a814ae1648e2bde5e521ef45af6d from peer 92cb785c165749eca49af5784756461c at {username='slave'} at 127.17.243.2:42935: session id = 92cb785c165749eca49af5784756461c-2078a814ae1648e2bde5e521ef45af6d
I20240502 01:58:45.208294 27332 tablet_copy_source_session.cc:205] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.209518 30211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2078a814ae1648e2bde5e521ef45af6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.211848 30211 tablet_copy_client.cc:783] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.212047 30211 tablet_copy_client.cc:647] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.213564 30211 tablet_copy_client.cc:523] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.213758 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:45.215145 30211 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:45.218580 27289 tablet_service.cc:1467] Processing CreateTablet for tablet af0156f7e3fa448ca2d409632cb0025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]), partition=
I20240502 01:58:45.218796 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0156f7e3fa448ca2d409632cb0025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.219053 27155 tablet_service.cc:1467] Processing CreateTablet for tablet af0156f7e3fa448ca2d409632cb0025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]), partition=
I20240502 01:58:45.219291 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0156f7e3fa448ca2d409632cb0025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.220260 30145 tablet_bootstrap.cc:492] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:45.220433 27425 tablet_service.cc:1467] Processing CreateTablet for tablet af0156f7e3fa448ca2d409632cb0025f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]), partition=
I20240502 01:58:45.220628 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0156f7e3fa448ca2d409632cb0025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.221205 30145 tablet_bootstrap.cc:654] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.221414 30147 tablet_bootstrap.cc:492] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:45.222318 30145 tablet_bootstrap.cc:492] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:45.222407 30145 ts_tablet_manager.cc:1367] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:45.222483 30147 tablet_bootstrap.cc:654] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.222462 29774 tablet_bootstrap.cc:492] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:45.222867 30145 raft_consensus.cc:348] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.222986 30145 raft_consensus.cc:374] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.223021 30145 raft_consensus.cc:729] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.223136 30145 consensus_queue.cc:260] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.223393 29774 tablet_bootstrap.cc:654] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.223475 30145 ts_tablet_manager.cc:1398] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:45.223851 30147 tablet_bootstrap.cc:492] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:45.223933 30147 ts_tablet_manager.cc:1367] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:45.224416 30147 raft_consensus.cc:348] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.224467 30211 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.224505 30147 raft_consensus.cc:374] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.224526 30147 raft_consensus.cc:729] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.224632 30147 consensus_queue.cc:260] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.224788 30211 tablet_bootstrap.cc:492] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: Bootstrap complete.
I20240502 01:58:45.224893 30147 ts_tablet_manager.cc:1398] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.224941 30211 ts_tablet_manager.cc:1367] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 01:58:45.225406 30211 raft_consensus.cc:348] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:45.225543 30211 raft_consensus.cc:729] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: LEARNER
I20240502 01:58:45.225661 30211 consensus_queue.cc:260] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:45.226069 30211 ts_tablet_manager.cc:1398] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.227010 29774 tablet_bootstrap.cc:492] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:45.227006 27332 tablet_copy_service.cc:342] P 1049b4c8783b48f39be22bd113ad349f: Request end of tablet copy session 92cb785c165749eca49af5784756461c-2078a814ae1648e2bde5e521ef45af6d received from {username='slave'} at 127.17.243.2:42935
I20240502 01:58:45.227077 27332 tablet_copy_service.cc:433] P 1049b4c8783b48f39be22bd113ad349f: ending tablet copy session 92cb785c165749eca49af5784756461c-2078a814ae1648e2bde5e521ef45af6d on tablet 2078a814ae1648e2bde5e521ef45af6d with peer 92cb785c165749eca49af5784756461c
I20240502 01:58:45.227093 29774 ts_tablet_manager.cc:1367] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:58:45.227442 29774 raft_consensus.cc:348] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.227540 29774 raft_consensus.cc:374] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.227576 29774 raft_consensus.cc:729] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.227682 29774 consensus_queue.cc:260] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.227959 29774 ts_tablet_manager.cc:1398] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:45.231321 29916 ts_tablet_manager.cc:902] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:45.231741 29916 tablet_copy_client.cc:308] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:45.232165 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet d136e006dd1048739156526af42fddd1 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:42087)
I20240502 01:58:45.232265 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet d136e006dd1048739156526af42fddd1 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:42087: session id = 9dd619a595e94004ac0f7f5a9054e7cc-d136e006dd1048739156526af42fddd1
I20240502 01:58:45.233132 24918 tablet_copy_source_session.cc:205] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.233865 29916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d136e006dd1048739156526af42fddd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.235508 29916 tablet_copy_client.cc:783] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.235644 29916 tablet_copy_client.cc:647] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.236860 29916 tablet_copy_client.cc:523] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.238003 29916 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:45.247373 30072 raft_consensus.cc:482] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.247473 30072 raft_consensus.cc:504] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.247848 30072 leader_election.cc:288] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:45.248163 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0156f7e3fa448ca2d409632cb0025f" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:45.248212 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0156f7e3fa448ca2d409632cb0025f" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:45.248322 27446 raft_consensus.cc:2454] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:45.248332 27177 raft_consensus.cc:2454] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:45.248602 27245 leader_election.cc:302] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:45.248971 30072 raft_consensus.cc:2781] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:45.249032 30072 raft_consensus.cc:482] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:45.249070 30072 raft_consensus.cc:3037] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.250123 30072 raft_consensus.cc:504] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.250433 30072 leader_election.cc:288] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:45.250849 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0156f7e3fa448ca2d409632cb0025f" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:45.250844 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0156f7e3fa448ca2d409632cb0025f" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:45.250942 27446 raft_consensus.cc:3037] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.250942 27177 raft_consensus.cc:3037] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.251781 27177 raft_consensus.cc:2454] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:45.251791 27446 raft_consensus.cc:2454] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:45.252100 27246 leader_election.cc:302] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:45.252257 30063 raft_consensus.cc:2781] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:45.252336 30063 raft_consensus.cc:686] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:45.252471 30063 consensus_queue.cc:237] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.254413 24485 catalog_manager.cc:5484] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:45.263476 29916 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.263751 29916 tablet_bootstrap.cc:492] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:45.263936 29916 ts_tablet_manager.cc:1367] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.004s
I20240502 01:58:45.264367 29916 raft_consensus.cc:348] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:45.264510 29916 raft_consensus.cc:729] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:45.264627 29916 consensus_queue.cc:260] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:45.265033 29916 ts_tablet_manager.cc:1398] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.265528 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-d136e006dd1048739156526af42fddd1 received from {username='slave'} at 127.17.243.5:42087
I20240502 01:58:45.265602 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-d136e006dd1048739156526af42fddd1 on tablet d136e006dd1048739156526af42fddd1 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:45.275807 27576 raft_consensus.cc:1203] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:45.282590 29098 raft_consensus.cc:482] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.282838 29098 raft_consensus.cc:504] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.283277 29098 leader_election.cc:288] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:45.283591 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8919df9a8a574882bad4995fc5df7e50" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:45.283607 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8919df9a8a574882bad4995fc5df7e50" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:45.283715 27311 raft_consensus.cc:2454] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:45.283721 27446 raft_consensus.cc:2454] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:45.283985 27113 leader_election.cc:302] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:45.284653 29098 raft_consensus.cc:2781] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:45.284713 29098 raft_consensus.cc:482] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:45.284749 29098 raft_consensus.cc:3037] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.285745 29098 raft_consensus.cc:504] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.286114 29098 leader_election.cc:288] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:45.286857 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8919df9a8a574882bad4995fc5df7e50" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:45.286975 27446 raft_consensus.cc:3037] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.287192 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8919df9a8a574882bad4995fc5df7e50" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:45.287292 27311 raft_consensus.cc:3037] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.287895 27446 raft_consensus.cc:2454] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:45.288000 27311 raft_consensus.cc:2454] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:45.288172 27113 leader_election.cc:302] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:45.288470 29098 raft_consensus.cc:2781] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:45.288551 29098 raft_consensus.cc:686] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:45.288673 29098 consensus_queue.cc:237] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.290823 24486 catalog_manager.cc:5484] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:45.302781 30184 ts_tablet_manager.cc:902] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:45.303202 30184 tablet_copy_client.cc:308] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:45.303607 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 93695e2165e142f79293ac14e4f4fb84 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:55633)
I20240502 01:58:45.303711 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 93695e2165e142f79293ac14e4f4fb84 from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:55633: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-93695e2165e142f79293ac14e4f4fb84
I20240502 01:58:45.304668 24918 tablet_copy_source_session.cc:205] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.305828 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93695e2165e142f79293ac14e4f4fb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.308012 30184 tablet_copy_client.cc:783] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.308141 30184 tablet_copy_client.cc:647] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.309701 30184 tablet_copy_client.cc:523] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.310786 30184 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:45.333212 30184 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.333484 30184 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:45.333654 30184 ts_tablet_manager.cc:1367] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20240502 01:58:45.334064 30184 raft_consensus.cc:348] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:45.334168 30184 raft_consensus.cc:729] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:45.334275 30184 consensus_queue.cc:260] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:45.334565 30184 ts_tablet_manager.cc:1398] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 01:58:45.335211 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-93695e2165e142f79293ac14e4f4fb84 received from {username='slave'} at 127.17.243.7:55633
I20240502 01:58:45.335290 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-93695e2165e142f79293ac14e4f4fb84 on tablet 93695e2165e142f79293ac14e4f4fb84 with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:45.335209 24763 consensus_queue.cc:237] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:45.336632 24897 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.336989 25034 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.337122 28393 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:45.337741 24698 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92382869887b43f9a23a2c0ea0012720. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:45.337841 28393 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.339371 30082 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:45.340341 25034 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:45.341044 24897 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:45.341852 24486 catalog_manager.cc:5484] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 4, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:45.356730 24897 raft_consensus.cc:1203] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:45.359107 27177 raft_consensus.cc:1203] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:45.366071 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:45.370146 27177 raft_consensus.cc:1261] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:45.370230 27446 raft_consensus.cc:1261] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:45.370597 30072 consensus_queue.cc:1035] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.370756 30063 consensus_queue.cc:1035] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.373389 24480 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]). Expected version 1 got 0
I20240502 01:58:45.373692 24485 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]). Expected version 1 got 0
I20240502 01:58:45.376264 30337 ts_tablet_manager.cc:902] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:45.376811 24485 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]). Expected version 1 got 0
I20240502 01:58:45.377115 29247 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:45.377291 30337 tablet_copy_client.cc:308] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:45.377485 29247 tablet.cc:1620] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:45.377882 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet c6186e8ffddc47a4a3b428fcd7ce19fb from peer 588bfc090a5247598cd5d3cee5ad2007 ({username='slave'} at 127.17.243.1:47327)
I20240502 01:58:45.378005 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet c6186e8ffddc47a4a3b428fcd7ce19fb from peer 588bfc090a5247598cd5d3cee5ad2007 at {username='slave'} at 127.17.243.1:47327: session id = 588bfc090a5247598cd5d3cee5ad2007-c6186e8ffddc47a4a3b428fcd7ce19fb
I20240502 01:58:45.378331 30345 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:45.379009 24785 tablet_copy_source_session.cc:205] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.380192 30345 tablet.cc:1620] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:45.380775 30337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6186e8ffddc47a4a3b428fcd7ce19fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.380908 24485 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]). Expected version 1 got 0
I20240502 01:58:45.381047 24480 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc] alter complete (version 1)
I20240502 01:58:45.383399 30337 tablet_copy_client.cc:783] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.383569 30337 tablet_copy_client.cc:647] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.385416 24486 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc]). Expected version 1 got 0
I20240502 01:58:45.385481 30338 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:45.386185 30338 tablet.cc:1620] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:45.386710 30337 tablet_copy_client.cc:523] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.388072 30337 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:45.392551 30337 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.392794 30337 tablet_bootstrap.cc:492] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap complete.
I20240502 01:58:45.392979 30337 ts_tablet_manager.cc:1367] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:58:45.393440 30337 raft_consensus.cc:348] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } }
I20240502 01:58:45.393574 30337 raft_consensus.cc:729] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: LEARNER
I20240502 01:58:45.393672 30337 consensus_queue.cc:260] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } }
I20240502 01:58:45.393954 30337 ts_tablet_manager.cc:1398] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.394418 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 588bfc090a5247598cd5d3cee5ad2007-c6186e8ffddc47a4a3b428fcd7ce19fb received from {username='slave'} at 127.17.243.1:47327
I20240502 01:58:45.394500 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 588bfc090a5247598cd5d3cee5ad2007-c6186e8ffddc47a4a3b428fcd7ce19fb on tablet c6186e8ffddc47a4a3b428fcd7ce19fb with peer 588bfc090a5247598cd5d3cee5ad2007
I20240502 01:58:45.483050 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:45.488337 30338 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:45.488667 30338 tablet.cc:1620] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:45.488814 30345 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:45.489096 30345 tablet.cc:1620] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:45.489475 29247 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:45.489732 29247 tablet.cc:1620] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:45.493364 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=02437dd305e840c5add13a596ec54bdc] alter complete (version 2)
I20240502 01:58:45.517886 28447 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.521320 28315 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.530092 25034 raft_consensus.cc:1203] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:45.536351 27446 raft_consensus.cc:1203] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:45.570649 27311 raft_consensus.cc:1203] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:45.595607 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:45.601083 30338 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:45.601498 30345 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:45.602388 29247 tablet.cc:1722] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:45.606370 24480 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=02437dd305e840c5add13a596ec54bdc] alter complete (version 3)
I20240502 01:58:45.625660 28315 raft_consensus.cc:1050] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:45.626212 28315 consensus_queue.cc:237] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:45.627362 24897 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.627437 27311 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.627468 27576 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.627810 30393 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.628018 28315 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.628175 28447 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.630154 28315 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.630582 27311 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.631006 27576 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.632939 24897 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.633689 27017 raft_consensus.cc:1050] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:45.633703 24480 catalog_manager.cc:5484] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.634142 27017 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:45.635937 24763 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:45.636083 27576 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:45.636301 27177 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:45.636888 27894 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:45.637230 27894 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:45.638176 27890 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:45.638963 27446 consensus_queue.cc:237] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:45.640020 24897 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.640062 27311 raft_consensus.cc:1261] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.640457 29244 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.640615 28447 consensus_queue.cc:1035] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.641234 28489 consensus_queue.cc:1035] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.641685 27890 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.642042 27177 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.643688 27576 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.644232 24486 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.644670 29244 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.647157 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a85e6a0806654abd81ca5661a2d019ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:45.647063 27311 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.647128 24480 catalog_manager.cc:5484] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.647444 24897 raft_consensus.cc:2932] T a85e6a0806654abd81ca5661a2d019ce P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.650329 25032 consensus_queue.cc:237] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:45.651782 27578 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:45.652118 24763 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.652328 27177 raft_consensus.cc:1261] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:45.652688 27017 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:45.652987 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet a85e6a0806654abd81ca5661a2d019ce with delete_type TABLET_DATA_TOMBSTONED (TS bb653087e8554c7591cf547dd969f3be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:45.654690 27890 consensus_queue.cc:1035] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:45.655588 28489 consensus_queue.cc:1035] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.656205 27894 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.659252 27576 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.660976 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 488816b0d5d542bcb5fa12ff7e70fd40 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:45.660876 27177 raft_consensus.cc:2932] T 488816b0d5d542bcb5fa12ff7e70fd40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.662583 24483 catalog_manager.cc:5484] T 488816b0d5d542bcb5fa12ff7e70fd40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 8 to 9, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.664019 30409 tablet_replica.cc:330] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:45.664206 30409 raft_consensus.cc:2229] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.664403 30409 raft_consensus.cc:2258] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.665414 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 488816b0d5d542bcb5fa12ff7e70fd40 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:45.665904 30412 tablet_replica.cc:330] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:45.665983 30409 ts_tablet_manager.cc:1875] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:45.666105 30412 raft_consensus.cc:2229] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.666584 30412 raft_consensus.cc:2258] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.667287 30412 ts_tablet_manager.cc:1875] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:45.669102 30409 ts_tablet_manager.cc:1888] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:45.669181 30409 log.cc:1196] T a85e6a0806654abd81ca5661a2d019ce P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/a85e6a0806654abd81ca5661a2d019ce
I20240502 01:58:45.669450 30412 ts_tablet_manager.cc:1888] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:58:45.669525 30412 log.cc:1196] T 488816b0d5d542bcb5fa12ff7e70fd40 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/488816b0d5d542bcb5fa12ff7e70fd40
I20240502 01:58:45.669687 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet a85e6a0806654abd81ca5661a2d019ce (table rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]) successfully deleted
I20240502 01:58:45.670326 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 488816b0d5d542bcb5fa12ff7e70fd40 (table rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]) successfully deleted
I20240502 01:58:45.693243 27446 raft_consensus.cc:1203] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:45.699492 24763 raft_consensus.cc:1203] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:45.707962 24485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:45.708101 24485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:45.710218 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet af0156f7e3fa448ca2d409632cb0025f
I20240502 01:58:45.710911 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet af0156f7e3fa448ca2d409632cb0025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:45 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:45.711035 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet af0156f7e3fa448ca2d409632cb0025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:45 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:45.711046 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet af0156f7e3fa448ca2d409632cb0025f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:45 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:45.711264 30454 tablet_replica.cc:330] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:45.711341 30456 tablet_replica.cc:330] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:45.711439 30454 raft_consensus.cc:2229] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:45.711463 30456 raft_consensus.cc:2229] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.711567 30456 raft_consensus.cc:2258] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.711652 30454 raft_consensus.cc:2258] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.712303 30454 ts_tablet_manager.cc:1875] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:45.712304 30456 ts_tablet_manager.cc:1875] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:45.712746 30455 tablet_replica.cc:330] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:45.712910 30455 raft_consensus.cc:2229] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.713037 30455 raft_consensus.cc:2258] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.713690 30455 ts_tablet_manager.cc:1875] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:45.714996 30456 ts_tablet_manager.cc:1888] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:45.715122 30456 log.cc:1196] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/af0156f7e3fa448ca2d409632cb0025f
I20240502 01:58:45.715453 30456 ts_tablet_manager.cc:1909] T af0156f7e3fa448ca2d409632cb0025f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:45.716205 30454 ts_tablet_manager.cc:1888] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:45.716223 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table_new_name [id=02437dd305e840c5add13a596ec54bdc]) successfully deleted
I20240502 01:58:45.716315 30454 log.cc:1196] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/af0156f7e3fa448ca2d409632cb0025f
I20240502 01:58:45.716552 30454 ts_tablet_manager.cc:1909] T af0156f7e3fa448ca2d409632cb0025f P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:45.717109 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table_new_name [id=02437dd305e840c5add13a596ec54bdc]) successfully deleted
I20240502 01:58:45.717391 30455 ts_tablet_manager.cc:1888] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:45.717473 30455 log.cc:1196] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/af0156f7e3fa448ca2d409632cb0025f
I20240502 01:58:45.717725 30455 ts_tablet_manager.cc:1909] T af0156f7e3fa448ca2d409632cb0025f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:45.718313 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet af0156f7e3fa448ca2d409632cb0025f (table rebalancer_dynamic_table_new_name [id=02437dd305e840c5add13a596ec54bdc]) successfully deleted
I20240502 01:58:45.739637 27177 raft_consensus.cc:1203] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:45.746327 24632 consensus_queue.cc:237] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:45.747640 24763 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:45.747812 24566 consensus_peers.cc:485] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 953a778a807a484b9e0376bdeeed8343. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:45.748065 28489 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.748158 24897 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.748623 28489 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.749825 29929 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:45.750464 24763 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:45.752765 24485 catalog_manager.cc:5484] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:45.757441 29579 raft_consensus.cc:1050] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:45.757840 29579 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:45.759783 27576 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:45.759826 24763 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:45.760663 24632 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:45.760859 27093 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:45.761205 27093 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:45.761484 27093 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:45.762652 24897 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:45.766961 29579 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:45.767386 27576 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:45.768353 24763 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:45.768891 24632 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:45.771028 24480 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 4 to 5, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:45.785593 27894 raft_consensus.cc:1050] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:45.786628 27894 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:45.788287 27311 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:45.788287 27177 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:45.788401 24896 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:45.788751 27017 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:45.788990 30398 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:45.789206 27894 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:45.791163 27890 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.791639 27311 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.792933 24896 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.793392 27177 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.796077 24480 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:45.797158 30520 consensus_queue.cc:1035] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.803380 30526 consensus_queue.cc:1035] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:45.812124 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:45.817384 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]), partition=
I20240502 01:58:45.817516 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]), partition=
I20240502 01:58:45.817633 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b38a8ac3e904a6dbb2223ecf062f48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.817709 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b38a8ac3e904a6dbb2223ecf062f48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.819890 30541 tablet_bootstrap.cc:492] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:45.820675 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]), partition=
I20240502 01:58:45.820920 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b38a8ac3e904a6dbb2223ecf062f48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.820988 30541 tablet_bootstrap.cc:654] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.821974 30540 tablet_bootstrap.cc:492] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:45.822104 30541 tablet_bootstrap.cc:492] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:45.822188 30541 ts_tablet_manager.cc:1367] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:45.822592 30541 raft_consensus.cc:348] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.822687 30541 raft_consensus.cc:374] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.822715 30541 raft_consensus.cc:729] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.822868 30540 tablet_bootstrap.cc:654] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.822835 30541 consensus_queue.cc:260] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.823136 30541 ts_tablet_manager.cc:1398] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.823861 30545 tablet_bootstrap.cc:492] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:45.824368 30544 ts_tablet_manager.cc:902] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:45.824976 30544 tablet_copy_client.cc:308] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:45.825732 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 92382869887b43f9a23a2c0ea0012720 from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:59521)
I20240502 01:58:45.825850 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 92382869887b43f9a23a2c0ea0012720 from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:59521: session id = 1049b4c8783b48f39be22bd113ad349f-92382869887b43f9a23a2c0ea0012720
I20240502 01:58:45.826325 30540 tablet_bootstrap.cc:492] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:45.826463 30540 ts_tablet_manager.cc:1367] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:45.826845 30540 raft_consensus.cc:348] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.826896 24785 tablet_copy_source_session.cc:205] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.827000 30540 raft_consensus.cc:374] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.827097 30540 raft_consensus.cc:729] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.827522 30540 consensus_queue.cc:260] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.827700 30545 tablet_bootstrap.cc:654] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.828090 30544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92382869887b43f9a23a2c0ea0012720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.828477 30540 ts_tablet_manager.cc:1398] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:45.830602 30545 tablet_bootstrap.cc:492] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:45.830803 30545 ts_tablet_manager.cc:1367] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:45.831089 30544 tablet_copy_client.cc:783] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.831247 30544 tablet_copy_client.cc:647] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.831696 30545 raft_consensus.cc:348] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.831897 30545 raft_consensus.cc:374] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.831977 30545 raft_consensus.cc:729] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.832166 30545 consensus_queue.cc:260] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.832329 30544 tablet_copy_client.cc:523] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.832569 30545 ts_tablet_manager.cc:1398] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:45.833408 30544 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:45.837522 25032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f"
dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37482
I20240502 01:58:45.837682 25032 raft_consensus.cc:595] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:45.843689 24632 raft_consensus.cc:1203] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:45.850893 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:45.856871 30544 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.857144 30544 tablet_bootstrap.cc:492] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:45.857323 30544 ts_tablet_manager.cc:1367] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.000s
I20240502 01:58:45.857779 30544 raft_consensus.cc:348] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:45.857909 30544 raft_consensus.cc:729] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:45.858028 30544 consensus_queue.cc:260] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:45.858405 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 60a07252321c4a0ebc16023c49015ffd (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.858623 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 60a07252321c4a0ebc16023c49015ffd (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.858723 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a07252321c4a0ebc16023c49015ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.858405 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.858891 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a07252321c4a0ebc16023c49015ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.858892 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.858850 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.859115 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.861124 30544 ts_tablet_manager.cc:1398] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:45.861193 30545 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:45.861307 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 9b066a49a2b444d3961cac30b5cb6adf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.861497 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b066a49a2b444d3961cac30b5cb6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.861569 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-92382869887b43f9a23a2c0ea0012720 received from {username='slave'} at 127.17.243.6:59521
I20240502 01:58:45.861634 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-92382869887b43f9a23a2c0ea0012720 on tablet 92382869887b43f9a23a2c0ea0012720 with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:45.862078 30545 tablet_bootstrap.cc:654] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.863068 30579 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:45.863435 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 60a07252321c4a0ebc16023c49015ffd (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.863720 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a07252321c4a0ebc16023c49015ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.863875 30579 tablet_bootstrap.cc:654] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.864492 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 9b066a49a2b444d3961cac30b5cb6adf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.864645 30545 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:45.864676 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b066a49a2b444d3961cac30b5cb6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.864720 30545 ts_tablet_manager.cc:1367] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:45.865131 30545 raft_consensus.cc:348] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.865235 30545 raft_consensus.cc:374] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.865262 30545 raft_consensus.cc:729] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.865377 30545 consensus_queue.cc:260] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.865661 30545 ts_tablet_manager.cc:1398] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:45.865820 30540 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:45.866055 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.866266 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.866658 30540 tablet_bootstrap.cc:654] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.866983 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 9b066a49a2b444d3961cac30b5cb6adf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:45.867205 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b066a49a2b444d3961cac30b5cb6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.868085 30540 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:45.868162 30540 ts_tablet_manager.cc:1367] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:45.868561 30540 raft_consensus.cc:348] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.868664 30540 raft_consensus.cc:374] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.868695 30540 raft_consensus.cc:729] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.868805 30540 consensus_queue.cc:260] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.869084 30540 ts_tablet_manager.cc:1398] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.869240 30540 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:45.869886 30540 tablet_bootstrap.cc:654] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.870774 30545 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:45.871548 30540 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:45.871666 30545 tablet_bootstrap.cc:654] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.871675 30540 ts_tablet_manager.cc:1367] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:45.872089 30540 raft_consensus.cc:348] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.872249 30540 raft_consensus.cc:374] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.872327 30540 raft_consensus.cc:729] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.872592 30540 consensus_queue.cc:260] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.873118 30580 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:45.873574 30540 ts_tablet_manager.cc:1398] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:45.874003 30580 tablet_bootstrap.cc:654] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.874264 30579 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:45.874347 30579 ts_tablet_manager.cc:1367] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240502 01:58:45.874956 30545 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:45.875030 30545 ts_tablet_manager.cc:1367] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:45.874743 30579 raft_consensus.cc:348] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.875360 30579 raft_consensus.cc:374] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.875392 30579 raft_consensus.cc:729] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.875399 30545 raft_consensus.cc:348] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.875525 30545 raft_consensus.cc:374] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.875499 30579 consensus_queue.cc:260] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.875560 30545 raft_consensus.cc:729] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.875669 30545 consensus_queue.cc:260] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.875949 30545 ts_tablet_manager.cc:1398] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.876161 30526 raft_consensus.cc:482] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.876242 30526 raft_consensus.cc:504] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.876327 30579 ts_tablet_manager.cc:1398] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:45.876581 30579 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:45.876669 30526 leader_election.cc:288] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:45.877374 30579 tablet_bootstrap.cc:654] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.877758 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b066a49a2b444d3961cac30b5cb6adf" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:45.877799 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b066a49a2b444d3961cac30b5cb6adf" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
W20240502 01:58:45.878185 27113 leader_election.cc:341] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:58:45.878278 27113 leader_election.cc:341] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:45.878311 27113 leader_election.cc:302] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc; no voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be
I20240502 01:58:45.878484 30520 raft_consensus.cc:2726] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:58:45.879565 30580 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:45.879699 30580 ts_tablet_manager.cc:1367] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:45.880105 30580 raft_consensus.cc:348] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.880213 30580 raft_consensus.cc:374] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.880239 30580 raft_consensus.cc:729] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.880342 30580 consensus_queue.cc:260] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.880650 30580 ts_tablet_manager.cc:1398] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.880844 30580 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:45.881750 30580 tablet_bootstrap.cc:654] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.882367 27867 raft_consensus.cc:482] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.882459 27867 raft_consensus.cc:504] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.882887 30580 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:45.882874 27867 leader_election.cc:288] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:45.882956 30580 ts_tablet_manager.cc:1367] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:45.883067 30579 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:45.883134 30579 ts_tablet_manager.cc:1367] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240502 01:58:45.883515 30580 raft_consensus.cc:348] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.883563 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a07252321c4a0ebc16023c49015ffd" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:45.883608 30580 raft_consensus.cc:374] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.883636 30580 raft_consensus.cc:729] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.883718 30579 raft_consensus.cc:348] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.883803 30579 raft_consensus.cc:374] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.883829 30579 raft_consensus.cc:729] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.883949 30579 consensus_queue.cc:260] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:45.883939 30580 consensus_queue.cc:260] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.884200 30580 ts_tablet_manager.cc:1398] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:45.883641 24763 raft_consensus.cc:2454] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:45.884550 27514 leader_election.cc:302] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:45.884692 27867 raft_consensus.cc:2781] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:45.884729 27867 raft_consensus.cc:482] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:45.884758 27867 raft_consensus.cc:3037] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.885567 27867 raft_consensus.cc:504] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.885862 27867 leader_election.cc:288] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:45.886183 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a07252321c4a0ebc16023c49015ffd" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:45.886282 24763 raft_consensus.cc:3037] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.886363 30579 ts_tablet_manager.cc:1398] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:45.887184 24763 raft_consensus.cc:2454] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:45.887856 27514 leader_election.cc:302] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:45.888028 27867 raft_consensus.cc:2781] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:45.888149 27867 raft_consensus.cc:686] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:45.888267 27867 consensus_queue.cc:237] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:45.888927 30579 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:45.890308 30579 tablet_bootstrap.cc:654] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:45.890758 24483 catalog_manager.cc:5484] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.892349 30579 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:45.892519 30579 ts_tablet_manager.cc:1367] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:45.893805 30579 raft_consensus.cc:348] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.894066 30579 raft_consensus.cc:374] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:45.894117 30579 raft_consensus.cc:729] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.894258 30579 consensus_queue.cc:260] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:45.894752 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a07252321c4a0ebc16023c49015ffd" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:45.894886 27446 raft_consensus.cc:2454] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:45.894977 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a07252321c4a0ebc16023c49015ffd" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:45.895076 27445 raft_consensus.cc:3037] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.895949 27445 raft_consensus.cc:2454] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:45.896350 30579 ts_tablet_manager.cc:1398] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:45.902469 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55824
I20240502 01:58:45.902642 24896 raft_consensus.cc:595] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:45.938074 28315 raft_consensus.cc:1050] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: attempting to promote NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 to VOTER
I20240502 01:58:45.938522 28315 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:45.939940 27578 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.939944 25032 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.940268 27311 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.940315 28315 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.940655 28315 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.940804 30393 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.942602 30393 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.943207 25032 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.943481 27311 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.944875 27578 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.945597 24483 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.945958 28447 raft_consensus.cc:1050] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:45.946329 28447 consensus_queue.cc:237] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:45.948040 27311 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.948513 30605 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.948824 27177 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.949038 24896 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:45.949690 28315 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.950018 28315 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:45.950117 27445 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:45.951221 27578 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.951773 30604 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.953073 25032 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.953439 30604 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.953902 30604 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.954478 27578 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.954504 28315 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.956054 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adc992deb712458aa793eaa11019de1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:45.956295 27177 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.956530 27311 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.956739 24480 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.958943 24486 catalog_manager.cc:5484] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.960592 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet adc992deb712458aa793eaa11019de1f with delete_type TABLET_DATA_TOMBSTONED (TS 1049b4c8783b48f39be22bd113ad349f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:45.960638 24896 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.960801 30454 tablet_replica.cc:330] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:45.960935 30454 raft_consensus.cc:2229] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.961064 30454 raft_consensus.cc:2258] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.961717 30454 ts_tablet_manager.cc:1875] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:45.963784 30454 ts_tablet_manager.cc:1888] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:45.963866 30454 log.cc:1196] T adc992deb712458aa793eaa11019de1f P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/adc992deb712458aa793eaa11019de1f
I20240502 01:58:45.964308 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet adc992deb712458aa793eaa11019de1f (table rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]) successfully deleted
I20240502 01:58:45.964358 25032 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:45.967525 27445 consensus_queue.cc:237] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:45.968564 27311 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.968662 24896 raft_consensus.cc:1261] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:45.969054 30604 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.969228 30605 consensus_queue.cc:1035] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:45.970943 28447 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.972092 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6c09e8f421d4ae2adf7c9b63121943c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:45.972379 27312 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.973209 24483 catalog_manager.cc:5484] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:45.974679 24896 raft_consensus.cc:2932] T b6c09e8f421d4ae2adf7c9b63121943c P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:45.975970 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c09e8f421d4ae2adf7c9b63121943c with delete_type TABLET_DATA_TOMBSTONED (TS 9dd619a595e94004ac0f7f5a9054e7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:45.976238 30456 tablet_replica.cc:330] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:45.976389 30456 raft_consensus.cc:2229] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:45.976527 30456 raft_consensus.cc:2258] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:45.977612 30456 ts_tablet_manager.cc:1875] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:45.979568 30456 ts_tablet_manager.cc:1888] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:45.979631 30456 log.cc:1196] T b6c09e8f421d4ae2adf7c9b63121943c P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/b6c09e8f421d4ae2adf7c9b63121943c
I20240502 01:58:45.980046 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet b6c09e8f421d4ae2adf7c9b63121943c (table rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]) successfully deleted
I20240502 01:58:45.980316 30605 raft_consensus.cc:482] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:45.980530 30605 raft_consensus.cc:504] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.980836 30605 leader_election.cc:288] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:45.981108 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b38a8ac3e904a6dbb2223ecf062f48b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:45.981216 24896 raft_consensus.cc:2454] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:45.981212 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b38a8ac3e904a6dbb2223ecf062f48b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:45.981369 27177 raft_consensus.cc:2454] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:45.981473 27380 leader_election.cc:302] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:45.981628 30605 raft_consensus.cc:2781] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:45.981681 30605 raft_consensus.cc:482] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:45.981707 30605 raft_consensus.cc:3037] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.982570 30605 raft_consensus.cc:504] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.982862 30605 leader_election.cc:288] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:45.983171 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b38a8ac3e904a6dbb2223ecf062f48b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:45.983201 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b38a8ac3e904a6dbb2223ecf062f48b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:45.983270 24896 raft_consensus.cc:3037] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.983302 27177 raft_consensus.cc:3037] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:45.983989 24896 raft_consensus.cc:2454] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:45.984153 27177 raft_consensus.cc:2454] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:45.984243 27380 leader_election.cc:302] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:45.984436 30605 raft_consensus.cc:2781] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:45.984534 30605 raft_consensus.cc:686] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:45.984647 30605 consensus_queue.cc:237] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:45.986425 24485 catalog_manager.cc:5484] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.012719 29574 raft_consensus.cc:1050] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:46.013142 29574 consensus_queue.cc:237] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:46.014604 27445 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.014604 24763 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.014657 25032 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.015022 29579 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.015226 29574 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.015430 30492 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.017282 29579 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.017765 27445 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.018101 24763 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.020356 25032 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.020942 24485 catalog_manager.cc:5484] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.055205 29574 raft_consensus.cc:1050] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:46.055629 29574 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:46.056962 24632 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:46.056967 27312 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:46.057096 27445 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:46.057396 29574 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:46.057607 30492 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:46.057808 29579 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:46.060163 30492 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.060546 27312 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.062259 24630 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.063913 27445 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.063951 24483 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index 7 to 8, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.069602 24896 consensus_queue.cc:237] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:46.070611 27445 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:46.070690 27312 raft_consensus.cc:1261] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:46.071012 29574 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:46.071190 29579 consensus_queue.cc:1035] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:46.073231 27093 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.073665 27446 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.074754 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b3edec06b3d8450a8cc3002be70c848a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:46.075081 27312 raft_consensus.cc:2932] T b3edec06b3d8450a8cc3002be70c848a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.076601 24485 catalog_manager.cc:5484] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 8 to 9, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.077140 27093 raft_consensus.cc:1050] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:46.077517 27093 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:46.078949 24630 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.079011 24763 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.079394 27446 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.079445 29579 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.079831 27093 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.080056 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet b3edec06b3d8450a8cc3002be70c848a with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:46.080338 27093 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.082243 30619 tablet_replica.cc:330] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:46.082226 29579 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.082379 30619 raft_consensus.cc:2229] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.082580 30619 raft_consensus.cc:2258] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.082720 24630 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.083925 30619 ts_tablet_manager.cc:1875] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.084095 27446 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.085070 24763 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.085467 24486 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 4 to 5, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.088644 30619 ts_tablet_manager.cc:1888] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:58:46.088708 30619 log.cc:1196] T b3edec06b3d8450a8cc3002be70c848a P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3edec06b3d8450a8cc3002be70c848a
I20240502 01:58:46.089196 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet b3edec06b3d8450a8cc3002be70c848a (table rebalance_test_table_3 [id=6b895318260e43ccadac5eebfe3ab194]) successfully deleted
I20240502 01:58:46.089783 24896 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:46.091285 27446 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:46.091384 24630 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:46.091634 29579 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:46.091827 27093 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:46.093309 29579 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.093986 24630 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.094089 27446 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.095249 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93695e2165e142f79293ac14e4f4fb84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:46.096446 24483 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:46.098775 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 93695e2165e142f79293ac14e4f4fb84 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:46.098999 30412 tablet_replica.cc:330] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:46.099123 30412 raft_consensus.cc:2229] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.099272 30412 raft_consensus.cc:2258] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.100474 30412 ts_tablet_manager.cc:1875] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.102782 30412 ts_tablet_manager.cc:1888] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:46.102855 30412 log.cc:1196] T 93695e2165e142f79293ac14e4f4fb84 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/93695e2165e142f79293ac14e4f4fb84
I20240502 01:58:46.103369 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 93695e2165e142f79293ac14e4f4fb84 (table rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]) successfully deleted
I20240502 01:58:46.117506 30520 raft_consensus.cc:482] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:46.117647 30520 raft_consensus.cc:504] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.118089 30520 leader_election.cc:288] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:46.118513 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8e4bca52cc47e89ad2b02d5035b1b7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:46.118618 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8e4bca52cc47e89ad2b02d5035b1b7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:46.118654 27578 raft_consensus.cc:2454] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:46.118731 27446 raft_consensus.cc:2454] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:46.119269 27113 leader_election.cc:302] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:46.119596 30520 raft_consensus.cc:2781] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:46.119653 30520 raft_consensus.cc:482] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:46.119690 30520 raft_consensus.cc:3037] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.119799 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:46.120779 30520 raft_consensus.cc:504] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.121146 30520 leader_election.cc:288] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:46.121760 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8e4bca52cc47e89ad2b02d5035b1b7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:46.121876 27446 raft_consensus.cc:3037] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.122048 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8e4bca52cc47e89ad2b02d5035b1b7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:46.122169 27578 raft_consensus.cc:3037] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.122862 27446 raft_consensus.cc:2454] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:46.123093 27578 raft_consensus.cc:2454] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:46.123189 27113 leader_election.cc:302] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:46.124370 30520 raft_consensus.cc:2781] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:46.124452 30520 raft_consensus.cc:686] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:46.124586 30520 consensus_queue.cc:237] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.126372 24896 raft_consensus.cc:1261] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.126814 24485 catalog_manager.cc:5484] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.127079 30605 consensus_queue.cc:1035] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.127761 27177 raft_consensus.cc:1261] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.128450 30605 consensus_queue.cc:1035] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.131707 24485 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]). Expected version 1 got 0
I20240502 01:58:46.133009 24485 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]). Expected version 1 got 0
I20240502 01:58:46.133396 24480 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]). Expected version 1 got 0
I20240502 01:58:46.133534 30646 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:46.133858 30646 tablet.cc:1620] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:46.134966 24480 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]). Expected version 1 got 0
I20240502 01:58:46.135596 24485 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940]). Expected version 1 got 0
I20240502 01:58:46.135890 30647 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:46.136382 30647 tablet.cc:1620] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:46.136771 30649 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:46.137696 30649 tablet.cc:1620] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:46.138675 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940] alter complete (version 1)
I20240502 01:58:46.161828 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55854
I20240502 01:58:46.162003 24896 raft_consensus.cc:595] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.176582 25032 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f"
dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37510
I20240502 01:58:46.176713 25032 raft_consensus.cc:595] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.198047 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55878
I20240502 01:58:46.198189 24896 raft_consensus.cc:595] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.199848 29574 raft_consensus.cc:979] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: : Instructing follower 92cb785c165749eca49af5784756461c to start an election
I20240502 01:58:46.199949 27093 raft_consensus.cc:1067] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Signalling peer 92cb785c165749eca49af5784756461c to start an election
I20240502 01:58:46.201170 24765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf"
dest_uuid: "92cb785c165749eca49af5784756461c"
from {username='slave'} at 127.17.243.3:56775
I20240502 01:58:46.201328 24765 raft_consensus.cc:482] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:46.201395 24765 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.202541 24765 raft_consensus.cc:504] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:46.202998 24765 leader_election.cc:288] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 2 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:46.203287 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:46.203425 24630 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.203646 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:46.203787 27578 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.204465 24630 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 2.
I20240502 01:58:46.204635 27578 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 2.
I20240502 01:58:46.205235 24700 leader_election.cc:302] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:46.205246 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:46.205346 24896 raft_consensus.cc:3032] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:46.205381 24896 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:46.205430 28393 raft_consensus.cc:2781] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:46.205524 28393 raft_consensus.cc:686] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:46.205543 24896 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:46.205667 28393 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:46.205724 24896 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.206707 24896 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 2.
I20240502 01:58:46.207716 24485 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 1 to 2, leader changed from f46bcd2bc124471981349b1e2726253b (127.17.243.3) to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 2 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.208865 27312 raft_consensus.cc:1203] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:46.210614 27093 raft_consensus.cc:1050] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:46.210796 27177 consensus_queue.cc:237] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:46.210978 27093 consensus_queue.cc:237] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.212352 25032 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.212833 30492 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.213238 27445 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.213487 27176 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.213652 30526 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:46.213955 30492 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:58:46.214246 27113 consensus_peers.cc:485] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2436f6b4344a2294be04f84f52f5ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:46.215126 27312 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.215513 30526 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:46.216087 30520 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.216567 27445 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.217536 24765 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.218425 29579 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.218917 27312 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.218971 24485 catalog_manager.cc:5484] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.222275 29579 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:46.222842 30715 ts_tablet_manager.cc:902] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:46.223363 30715 tablet_copy_client.cc:308] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:46.224257 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 953a778a807a484b9e0376bdeeed8343 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:52035)
I20240502 01:58:46.224361 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 953a778a807a484b9e0376bdeeed8343 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:52035: session id = 9dd619a595e94004ac0f7f5a9054e7cc-953a778a807a484b9e0376bdeeed8343
I20240502 01:58:46.224781 27176 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:46.225205 24652 tablet_copy_source_session.cc:205] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:46.226469 30715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953a778a807a484b9e0376bdeeed8343. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.226377 25032 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:46.228209 24765 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:46.228372 30715 tablet_copy_client.cc:783] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:46.228514 30715 tablet_copy_client.cc:647] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:46.229111 24480 catalog_manager.cc:5484] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:46.232218 30715 tablet_copy_client.cc:523] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:46.233428 28393 raft_consensus.cc:482] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:46.233538 30715 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:46.233518 28393 raft_consensus.cc:504] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:46.233865 28393 leader_election.cc:288] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:46.234158 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b066a49a2b444d3961cac30b5cb6adf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:46.234272 27176 raft_consensus.cc:2454] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:46.234282 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b066a49a2b444d3961cac30b5cb6adf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:46.234407 27578 raft_consensus.cc:2454] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:46.234586 24700 leader_election.cc:302] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:46.234989 28393 raft_consensus.cc:2781] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:46.235060 28393 raft_consensus.cc:482] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:46.235100 28393 raft_consensus.cc:3037] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.236093 28393 raft_consensus.cc:504] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:46.236486 28393 leader_election.cc:288] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:46.237017 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b066a49a2b444d3961cac30b5cb6adf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:46.237123 27578 raft_consensus.cc:3037] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.237217 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b066a49a2b444d3961cac30b5cb6adf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:46.237318 27176 raft_consensus.cc:3037] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.238054 27578 raft_consensus.cc:2454] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:46.238118 27176 raft_consensus.cc:2454] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:46.238337 24701 leader_election.cc:302] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:46.238582 28393 raft_consensus.cc:2781] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:46.238683 28393 raft_consensus.cc:686] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:46.238821 28393 consensus_queue.cc:237] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:46.239152 25032 consensus_queue.cc:237] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:46.240839 24896 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:46.240936 24630 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:58:46.240981 24965 consensus_peers.cc:485] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61ad1a07bc7a4fbcb901792d138e2843. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:46.241245 27017 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:46.241432 27890 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:46.241694 24480 catalog_manager.cc:5484] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.242182 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:46.244086 30072 raft_consensus.cc:1050] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f: attempting to promote NON_VOTER 92cb785c165749eca49af5784756461c to VOTER
I20240502 01:58:46.244339 27894 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:46.244570 30072 consensus_queue.cc:237] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:46.244982 24896 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:46.246346 27445 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:46.246613 27576 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:46.247972 30072 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:46.249013 24480 catalog_manager.cc:5484] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:46.249239 24765 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:46.253715 24630 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:46.255102 30738 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:46.255367 30739 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:46.258574 30646 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:46.258503 30738 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.258875 30646 tablet.cc:1620] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:46.258854 24765 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.259399 30647 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:46.259698 30647 tablet.cc:1620] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:46.261540 24480 catalog_manager.cc:5484] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.263609 27445 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.266052 27576 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.266454 30649 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:46.266741 30649 tablet.cc:1620] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:46.268033 30715 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:46.268343 30715 tablet_bootstrap.cc:492] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:46.268582 30715 ts_tablet_manager.cc:1367] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.005s
I20240502 01:58:46.268934 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=f99986c242d04189ad9a285881bc5940] alter complete (version 2)
I20240502 01:58:46.269788 27312 consensus_queue.cc:237] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:46.270897 30715 raft_consensus.cc:348] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:46.271298 24765 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:46.271716 30736 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:46.272141 27576 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:46.272470 30739 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:46.271085 30715 raft_consensus.cc:729] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:46.273038 30715 consensus_queue.cc:260] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:46.273394 30739 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.274272 30715 ts_tablet_manager.cc:1398] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:46.275374 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2078a814ae1648e2bde5e521ef45af6d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:46.275703 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-953a778a807a484b9e0376bdeeed8343 received from {username='slave'} at 127.17.243.5:52035
I20240502 01:58:46.275514 24765 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.275779 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-953a778a807a484b9e0376bdeeed8343 on tablet 953a778a807a484b9e0376bdeeed8343 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:46.278136 24485 catalog_manager.cc:5484] T 2078a814ae1648e2bde5e521ef45af6d P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.278865 27578 raft_consensus.cc:2932] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:46.285334 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 2078a814ae1648e2bde5e521ef45af6d with delete_type TABLET_DATA_TOMBSTONED (TS 1cc0a6338e0a467dbdc6efb3e67a6a09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:46.287122 30754 tablet_replica.cc:330] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:46.287366 30754 raft_consensus.cc:2229] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.287628 30754 raft_consensus.cc:2258] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.287815 27894 raft_consensus.cc:979] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: : Instructing follower f46bcd2bc124471981349b1e2726253b to start an election
I20240502 01:58:46.287907 27894 raft_consensus.cc:1067] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Signalling peer f46bcd2bc124471981349b1e2726253b to start an election
I20240502 01:58:46.288461 24896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
from {username='slave'} at 127.17.243.4:50213
I20240502 01:58:46.288605 24896 raft_consensus.cc:482] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:46.288669 24896 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.288694 30754 ts_tablet_manager.cc:1875] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.289655 24896 raft_consensus.cc:504] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.290390 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:46.290513 25032 raft_consensus.cc:3032] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:46.290565 25032 raft_consensus.cc:729] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:46.290740 25032 consensus_queue.cc:260] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.290869 30754 ts_tablet_manager.cc:1888] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:46.290939 30754 log.cc:1196] T 2078a814ae1648e2bde5e521ef45af6d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/2078a814ae1648e2bde5e521ef45af6d
I20240502 01:58:46.290963 25032 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.291505 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 2078a814ae1648e2bde5e521ef45af6d (table rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]) successfully deleted
I20240502 01:58:46.291833 25032 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 2.
I20240502 01:58:46.292306 24896 leader_election.cc:288] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 2 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:46.292757 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:46.292887 27177 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.292879 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7501639064035a570b3b4c1cf083f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:46.293004 27312 raft_consensus.cc:3037] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.293782 27177 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 2.
I20240502 01:58:46.293901 27312 raft_consensus.cc:2454] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 2.
I20240502 01:58:46.294178 24832 leader_election.cc:302] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:46.294342 29579 raft_consensus.cc:2781] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:46.294433 29579 raft_consensus.cc:686] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 2 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:46.294564 29579 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.296440 24479 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 1 to 2, leader changed from e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 2 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.327479 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55888
I20240502 01:58:46.327607 24896 raft_consensus.cc:595] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.341522 30591 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.341658 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d136e006dd1048739156526af42fddd1"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55910
I20240502 01:58:46.341778 24896 raft_consensus.cc:595] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.347072 27867 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.361692 28393 raft_consensus.cc:1050] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 to VOTER
I20240502 01:58:46.362147 28393 consensus_queue.cc:237] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } }
I20240502 01:58:46.363451 27312 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:46.363911 27445 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:46.364320 30821 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:46.364796 24630 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:46.365181 30820 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:46.365630 30820 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:46.367449 30821 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.368415 24630 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.370061 27445 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.370878 24479 catalog_manager.cc:5484] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.371954 27312 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.374508 24765 consensus_queue.cc:237] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } }
I20240502 01:58:46.375759 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:46.375819 24630 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:46.375819 27445 raft_consensus.cc:1261] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:46.376370 30569 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:46.376778 30820 consensus_queue.cc:1035] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:46.378428 30569 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.379901 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6186e8ffddc47a4a3b428fcd7ce19fb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:46.380192 27445 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.381069 24630 raft_consensus.cc:2932] T c6186e8ffddc47a4a3b428fcd7ce19fb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:46.381202 24486 catalog_manager.cc:5484] T c6186e8ffddc47a4a3b428fcd7ce19fb P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.383426 30646 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:46.384212 30647 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:46.386888 30649 tablet.cc:1722] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:46.389391 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=f99986c242d04189ad9a285881bc5940] alter complete (version 3)
I20240502 01:58:46.391408 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet c6186e8ffddc47a4a3b428fcd7ce19fb with delete_type TABLET_DATA_TOMBSTONED (TS 1049b4c8783b48f39be22bd113ad349f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:46.392135 30454 tablet_replica.cc:330] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:46.392289 30454 raft_consensus.cc:2229] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.392452 30454 raft_consensus.cc:2258] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.393406 30454 ts_tablet_manager.cc:1875] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.395691 30454 ts_tablet_manager.cc:1888] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:46.395758 30454 log.cc:1196] T c6186e8ffddc47a4a3b428fcd7ce19fb P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/c6186e8ffddc47a4a3b428fcd7ce19fb
I20240502 01:58:46.396271 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet c6186e8ffddc47a4a3b428fcd7ce19fb (table rebalancer_extra_table_4 [id=ac42855140f14e698243b2c8ccb36cf4]) successfully deleted
I20240502 01:58:46.409382 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55930
I20240502 01:58:46.409523 24897 raft_consensus.cc:595] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.432924 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d136e006dd1048739156526af42fddd1"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55950
I20240502 01:58:46.433084 24897 raft_consensus.cc:595] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.474689 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55972
I20240502 01:58:46.474840 24897 raft_consensus.cc:595] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.491178 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d136e006dd1048739156526af42fddd1"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55992
I20240502 01:58:46.491308 24897 raft_consensus.cc:595] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.493316 24485 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:46.493468 24485 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:46.495893 24485 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 6b38a8ac3e904a6dbb2223ecf062f48b
I20240502 01:58:46.496434 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b38a8ac3e904a6dbb2223ecf062f48b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:46 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:46.496567 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b38a8ac3e904a6dbb2223ecf062f48b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:46 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:46.496780 30754 tablet_replica.cc:330] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:46.496881 30921 tablet_replica.cc:330] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:46.496930 30754 raft_consensus.cc:2229] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:46.497009 30921 raft_consensus.cc:2229] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.497113 30921 raft_consensus.cc:2258] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.497133 30754 raft_consensus.cc:2258] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.497303 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b38a8ac3e904a6dbb2223ecf062f48b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:46 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:46.497624 30922 tablet_replica.cc:330] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:46.497709 30921 ts_tablet_manager.cc:1875] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:46.497757 30922 raft_consensus.cc:2229] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.497874 30922 raft_consensus.cc:2258] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.498409 30754 ts_tablet_manager.cc:1875] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:46.498453 30922 ts_tablet_manager.cc:1875] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:46.500353 30922 ts_tablet_manager.cc:1888] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:46.500422 30922 log.cc:1196] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/6b38a8ac3e904a6dbb2223ecf062f48b
I20240502 01:58:46.500485 30921 ts_tablet_manager.cc:1888] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:46.500485 30754 ts_tablet_manager.cc:1888] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:46.500564 30921 log.cc:1196] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b38a8ac3e904a6dbb2223ecf062f48b
I20240502 01:58:46.500564 30754 log.cc:1196] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/6b38a8ac3e904a6dbb2223ecf062f48b
I20240502 01:58:46.500669 30922 ts_tablet_manager.cc:1909] T 6b38a8ac3e904a6dbb2223ecf062f48b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:46.500802 30754 ts_tablet_manager.cc:1909] T 6b38a8ac3e904a6dbb2223ecf062f48b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:46.500834 30921 ts_tablet_manager.cc:1909] T 6b38a8ac3e904a6dbb2223ecf062f48b P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:46.501361 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table_new_name [id=f99986c242d04189ad9a285881bc5940]) successfully deleted
I20240502 01:58:46.501622 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table_new_name [id=f99986c242d04189ad9a285881bc5940]) successfully deleted
I20240502 01:58:46.501912 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 6b38a8ac3e904a6dbb2223ecf062f48b (table rebalancer_dynamic_table_new_name [id=f99986c242d04189ad9a285881bc5940]) successfully deleted
I20240502 01:58:46.512861 30526 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.516538 30520 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.520612 29574 raft_consensus.cc:979] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: : Instructing follower e0b9e7df4e8543fa9b8e3ffc24330657 to start an election
I20240502 01:58:46.520701 29574 raft_consensus.cc:1067] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Signalling peer e0b9e7df4e8543fa9b8e3ffc24330657 to start an election
I20240502 01:58:46.521701 25032 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e"
dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
from {username='slave'} at 127.17.243.3:55137
I20240502 01:58:46.521853 25032 raft_consensus.cc:482] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:46.521914 25032 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.522727 25032 raft_consensus.cc:504] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:46.523129 25032 leader_election.cc:288] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 2 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:46.523514 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:46.523530 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:46.523648 27445 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.523653 24765 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.524454 24765 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 2.
I20240502 01:58:46.524469 27445 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 2.
I20240502 01:58:46.524722 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89580549c13d4c1ebdb2a5d24da2a01e" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:46.524761 24966 leader_election.cc:302] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:46.524834 24897 raft_consensus.cc:3032] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:46.524880 24897 raft_consensus.cc:729] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:46.524911 27017 raft_consensus.cc:2781] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:46.525012 27017 raft_consensus.cc:686] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:46.525053 24897 consensus_queue.cc:260] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:46.525135 27017 consensus_queue.cc:237] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:46.525246 24897 raft_consensus.cc:3037] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.526228 24897 raft_consensus.cc:2454] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 2.
I20240502 01:58:46.527060 24483 catalog_manager.cc:5484] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 1 to 2, leader changed from f46bcd2bc124471981349b1e2726253b (127.17.243.3) to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 2 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.597661 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:46.602793 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 080e8c628aa948c0a4d5798936d089ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]), partition=
I20240502 01:58:46.602804 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 080e8c628aa948c0a4d5798936d089ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]), partition=
I20240502 01:58:46.602804 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 080e8c628aa948c0a4d5798936d089ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]), partition=
I20240502 01:58:46.603065 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080e8c628aa948c0a4d5798936d089ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.603065 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080e8c628aa948c0a4d5798936d089ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.603075 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080e8c628aa948c0a4d5798936d089ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.604904 30925 tablet_bootstrap.cc:492] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:46.605249 30926 tablet_bootstrap.cc:492] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:46.605836 30925 tablet_bootstrap.cc:654] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.606230 30926 tablet_bootstrap.cc:654] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.607008 30927 tablet_bootstrap.cc:492] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:46.607383 30925 tablet_bootstrap.cc:492] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:46.607482 30925 ts_tablet_manager.cc:1367] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:46.607950 30927 tablet_bootstrap.cc:654] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.607914 30925 raft_consensus.cc:348] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.608021 30925 raft_consensus.cc:374] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.608057 30925 raft_consensus.cc:729] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.608179 30925 consensus_queue.cc:260] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.608486 30925 ts_tablet_manager.cc:1398] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:46.609301 30926 tablet_bootstrap.cc:492] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:46.609385 30926 ts_tablet_manager.cc:1367] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:46.610213 30926 raft_consensus.cc:348] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.610327 30926 raft_consensus.cc:374] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.610361 30926 raft_consensus.cc:729] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.610499 30926 consensus_queue.cc:260] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.611248 30926 ts_tablet_manager.cc:1398] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:46.612581 30927 tablet_bootstrap.cc:492] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:46.612731 30927 ts_tablet_manager.cc:1367] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:46.613365 30927 raft_consensus.cc:348] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.613581 30927 raft_consensus.cc:374] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.613691 30927 raft_consensus.cc:729] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.614023 30927 consensus_queue.cc:260] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.614487 30927 ts_tablet_manager.cc:1398] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:46.631170 27177 raft_consensus.cc:1203] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:46.634652 30393 raft_consensus.cc:482] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:46.634750 30393 raft_consensus.cc:504] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.635170 30393 leader_election.cc:288] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:46.635720 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080e8c628aa948c0a4d5798936d089ee" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:46.635823 29579 raft_consensus.cc:979] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: : Instructing follower e0b9e7df4e8543fa9b8e3ffc24330657 to start an election
I20240502 01:58:46.635860 27312 raft_consensus.cc:2454] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:46.635938 29579 raft_consensus.cc:1067] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Signalling peer e0b9e7df4e8543fa9b8e3ffc24330657 to start an election
I20240502 01:58:46.635982 27578 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:46.636137 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080e8c628aa948c0a4d5798936d089ee" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:46.636277 24765 raft_consensus.cc:2454] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:46.636545 25032 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d136e006dd1048739156526af42fddd1"
dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
from {username='slave'} at 127.17.243.3:55137
I20240502 01:58:46.636538 30820 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:46.636590 27377 leader_election.cc:302] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:46.636662 25032 raft_consensus.cc:482] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:46.636718 25032 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.636740 30393 raft_consensus.cc:2781] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:46.636804 30393 raft_consensus.cc:482] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:46.636848 30393 raft_consensus.cc:3037] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.637595 30393 raft_consensus.cc:504] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.637600 25032 raft_consensus.cc:504] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.638024 25032 leader_election.cc:288] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 2 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:46.638283 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080e8c628aa948c0a4d5798936d089ee" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:46.638446 27312 raft_consensus.cc:3037] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.638491 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:46.638613 24897 raft_consensus.cc:3032] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:46.638597 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:46.638653 24897 raft_consensus.cc:729] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:46.638702 27177 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.638792 24897 consensus_queue.cc:260] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.638996 24897 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.639375 27312 raft_consensus.cc:2454] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:46.639600 27177 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 2.
I20240502 01:58:46.639823 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d136e006dd1048739156526af42fddd1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:46.639859 27377 leader_election.cc:302] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:46.639950 24765 raft_consensus.cc:3037] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:46.640005 24897 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 2.
I20240502 01:58:46.640187 28447 raft_consensus.cc:2781] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:46.640269 28447 raft_consensus.cc:686] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:46.640374 24966 leader_election.cc:302] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:46.640388 28447 consensus_queue.cc:237] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:46.640554 30398 raft_consensus.cc:2781] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:46.640632 30398 raft_consensus.cc:686] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:46.640755 30398 consensus_queue.cc:237] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:46.640939 24765 raft_consensus.cc:2454] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 2.
I20240502 01:58:46.643005 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080e8c628aa948c0a4d5798936d089ee" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:46.643095 24765 raft_consensus.cc:3037] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.643425 24485 catalog_manager.cc:5484] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 1 to 2, leader changed from f46bcd2bc124471981349b1e2726253b (127.17.243.3) to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 2 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.643788 24479 catalog_manager.cc:5484] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:46.643886 24765 raft_consensus.cc:2454] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:46.637893 30393 leader_election.cc:288] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:46.691946 25032 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:46.692391 27093 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:46.703647 30939 ts_tablet_manager.cc:902] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:46.704171 30939 tablet_copy_client.cc:308] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.5:40239
I20240502 01:58:46.704864 27182 tablet_copy_service.cc:140] P 9dd619a595e94004ac0f7f5a9054e7cc: Received BeginTabletCopySession request for tablet 4f2436f6b4344a2294be04f84f52f5ff from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:55443)
I20240502 01:58:46.705000 27182 tablet_copy_service.cc:161] P 9dd619a595e94004ac0f7f5a9054e7cc: Beginning new tablet copy session on tablet 4f2436f6b4344a2294be04f84f52f5ff from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:55443: session id = f46bcd2bc124471981349b1e2726253b-4f2436f6b4344a2294be04f84f52f5ff
I20240502 01:58:46.706296 27182 tablet_copy_source_session.cc:205] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:46.707825 30939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2436f6b4344a2294be04f84f52f5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.710464 30939 tablet_copy_client.cc:783] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:46.710711 30939 tablet_copy_client.cc:647] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:46.711148 30820 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.713104 30939 tablet_copy_client.cc:523] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:46.715101 30821 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.719118 30939 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:46.722746 30939 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:46.722970 30939 tablet_bootstrap.cc:492] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:46.723136 30939 ts_tablet_manager.cc:1367] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:46.723580 30939 raft_consensus.cc:348] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:46.723690 30939 raft_consensus.cc:729] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:46.724133 30939 consensus_queue.cc:260] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:46.724205 24630 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:46.724749 30939 ts_tablet_manager.cc:1398] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:46.724684 30821 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:46.725386 27182 tablet_copy_service.cc:342] P 9dd619a595e94004ac0f7f5a9054e7cc: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-4f2436f6b4344a2294be04f84f52f5ff received from {username='slave'} at 127.17.243.3:55443
I20240502 01:58:46.725497 27182 tablet_copy_service.cc:433] P 9dd619a595e94004ac0f7f5a9054e7cc: ending tablet copy session f46bcd2bc124471981349b1e2726253b-4f2436f6b4344a2294be04f84f52f5ff on tablet 4f2436f6b4344a2294be04f84f52f5ff with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:46.728442 24897 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:46.728979 28393 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:46.731110 24765 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:46.732300 27578 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:46.732403 27177 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:46.732748 24632 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:46.733870 30569 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:46.734534 30569 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:46.734979 29574 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:46.740864 27312 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:46.741562 29574 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:46.744581 30569 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:46.745211 24632 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:46.746325 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44be529dec974eac8419efe9c19b30bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:46.747207 27578 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:46.750523 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:46.754808 24483 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 2 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:46.755163 24765 raft_consensus.cc:1261] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.755259 27312 raft_consensus.cc:1261] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.755872 30393 consensus_queue.cc:1035] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.756188 30393 consensus_queue.cc:1035] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:46.757670 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 44be529dec974eac8419efe9c19b30bf with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:46.757963 30921 tablet_replica.cc:330] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:46.758096 30921 raft_consensus.cc:2229] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.758196 30921 raft_consensus.cc:2258] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.758883 30921 ts_tablet_manager.cc:1875] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.761062 30646 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:46.761147 30921 ts_tablet_manager.cc:1888] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:46.761209 30921 log.cc:1196] T 44be529dec974eac8419efe9c19b30bf P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/44be529dec974eac8419efe9c19b30bf
I20240502 01:58:46.761402 30646 tablet.cc:1620] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:46.762145 24485 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]). Expected version 1 got 0
I20240502 01:58:46.762244 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 44be529dec974eac8419efe9c19b30bf (table rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]) successfully deleted
I20240502 01:58:46.762269 30715 ts_tablet_manager.cc:902] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:46.762706 30715 tablet_copy_client.cc:308] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:46.763089 24483 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]). Expected version 1 got 0
I20240502 01:58:46.763192 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 61ad1a07bc7a4fbcb901792d138e2843 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:54405)
I20240502 01:58:46.763278 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 61ad1a07bc7a4fbcb901792d138e2843 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:54405: session id = 9dd619a595e94004ac0f7f5a9054e7cc-61ad1a07bc7a4fbcb901792d138e2843
I20240502 01:58:46.764134 25054 tablet_copy_source_session.cc:205] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:46.764396 24485 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]). Expected version 1 got 0
I20240502 01:58:46.765172 30715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ad1a07bc7a4fbcb901792d138e2843. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.767309 30988 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:46.767755 30988 tablet.cc:1620] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:46.767866 24480 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]). Expected version 1 got 0
I20240502 01:58:46.768280 24483 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b]). Expected version 1 got 0
I20240502 01:58:46.768251 30983 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:46.768522 30983 tablet.cc:1620] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:46.769450 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b] alter complete (version 1)
I20240502 01:58:46.769647 30715 tablet_copy_client.cc:783] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:46.769841 30715 tablet_copy_client.cc:647] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:46.772071 30715 tablet_copy_client.cc:523] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:46.775617 30715 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:46.795871 30715 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:46.796128 30715 tablet_bootstrap.cc:492] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:46.796290 30715 ts_tablet_manager.cc:1367] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.003s
I20240502 01:58:46.796662 30715 raft_consensus.cc:348] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:46.796767 30715 raft_consensus.cc:729] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:46.796856 30715 consensus_queue.cc:260] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:46.797142 30715 ts_tablet_manager.cc:1398] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:46.797644 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-61ad1a07bc7a4fbcb901792d138e2843 received from {username='slave'} at 127.17.243.5:54405
I20240502 01:58:46.797724 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-61ad1a07bc7a4fbcb901792d138e2843 on tablet 61ad1a07bc7a4fbcb901792d138e2843 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:46.806685 27177 consensus_queue.cc:237] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:46.808055 27312 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:46.808262 27578 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:46.808360 27113 consensus_peers.cc:485] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c28e5d1164e4dfb997f2d7fec77429e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:46.808460 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:46.808910 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:46.810148 30520 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.810768 27312 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.810989 27578 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.813601 24483 catalog_manager.cc:5484] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:46.825486 24480 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:46.831501 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 334184346780410a9f7efe298ae4d5e1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.831811 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334184346780410a9f7efe298ae4d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.832103 27557 tablet_service.cc:1467] Processing CreateTablet for tablet c97e5f78b03245cc86f70b1d50fcf723 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.832326 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97e5f78b03245cc86f70b1d50fcf723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.834059 31020 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:46.834195 27156 tablet_service.cc:1467] Processing CreateTablet for tablet c97e5f78b03245cc86f70b1d50fcf723 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.834195 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 334184346780410a9f7efe298ae4d5e1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.834254 27290 tablet_service.cc:1467] Processing CreateTablet for tablet c97e5f78b03245cc86f70b1d50fcf723 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.834450 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334184346780410a9f7efe298ae4d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.834460 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97e5f78b03245cc86f70b1d50fcf723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.834509 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97e5f78b03245cc86f70b1d50fcf723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.835175 31020 tablet_bootstrap.cc:654] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.836426 30925 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:46.836587 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 334184346780410a9f7efe298ae4d5e1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.836798 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334184346780410a9f7efe298ae4d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.836800 31023 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:46.837167 31022 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:46.837417 30925 tablet_bootstrap.cc:654] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.837713 31023 tablet_bootstrap.cc:654] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.837985 31022 tablet_bootstrap.cc:654] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.838431 27557 tablet_service.cc:1467] Processing CreateTablet for tablet b2e41964ae354b4d92385663ae940bc0 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.838641 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e41964ae354b4d92385663ae940bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.838773 31023 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:46.838846 31023 ts_tablet_manager.cc:1367] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:46.838945 27424 tablet_service.cc:1467] Processing CreateTablet for tablet b2e41964ae354b4d92385663ae940bc0 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.839228 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e41964ae354b4d92385663ae940bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.839249 31023 raft_consensus.cc:348] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:46.839347 31023 raft_consensus.cc:374] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.839377 31023 raft_consensus.cc:729] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.839504 31023 consensus_queue.cc:260] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:46.839802 31023 ts_tablet_manager.cc:1398] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:46.840370 30925 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:46.840448 30925 ts_tablet_manager.cc:1367] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:46.840864 30925 raft_consensus.cc:348] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:46.840958 30925 raft_consensus.cc:374] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.840989 30925 raft_consensus.cc:729] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.841091 30925 consensus_queue.cc:260] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:46.841312 30927 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:46.841344 30925 ts_tablet_manager.cc:1398] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:46.841516 30925 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:46.842057 31022 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:46.842139 31022 ts_tablet_manager.cc:1367] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:58:46.842248 30927 tablet_bootstrap.cc:654] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.842358 30925 tablet_bootstrap.cc:654] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.842658 24610 tablet_service.cc:1467] Processing CreateTablet for tablet b2e41964ae354b4d92385663ae940bc0 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:46.842885 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e41964ae354b4d92385663ae940bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:46.843130 30821 raft_consensus.cc:1050] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER 1049b4c8783b48f39be22bd113ad349f to VOTER
I20240502 01:58:46.843295 30927 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:46.843364 30927 ts_tablet_manager.cc:1367] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:46.843544 30821 consensus_queue.cc:237] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:46.843776 30927 raft_consensus.cc:348] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.843889 30927 raft_consensus.cc:374] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.843922 30927 raft_consensus.cc:729] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.844033 30927 consensus_queue.cc:260] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.844285 30927 ts_tablet_manager.cc:1398] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:46.844930 31022 raft_consensus.cc:348] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.845062 31022 raft_consensus.cc:374] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.845108 31022 raft_consensus.cc:729] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.845393 30925 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:46.845482 30925 ts_tablet_manager.cc:1367] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:46.845770 24897 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.845880 30925 raft_consensus.cc:348] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.846000 30925 raft_consensus.cc:374] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.846035 30925 raft_consensus.cc:729] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.846155 30925 consensus_queue.cc:260] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.846249 30569 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.846413 30925 ts_tablet_manager.cc:1398] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:46.846464 31022 consensus_queue.cc:260] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.847332 31022 ts_tablet_manager.cc:1398] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:46.847481 31022 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:46.848104 25032 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.848302 31022 tablet_bootstrap.cc:654] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.848496 30569 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.848922 27312 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:46.849452 31020 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:46.849556 31020 ts_tablet_manager.cc:1367] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:46.849797 30569 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:46.850085 31020 raft_consensus.cc:348] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.850208 31020 raft_consensus.cc:374] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.850260 31020 raft_consensus.cc:729] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.850966 31020 consensus_queue.cc:260] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.851845 31020 ts_tablet_manager.cc:1398] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:46.852069 31020 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:46.852710 30820 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.853447 31020 tablet_bootstrap.cc:654] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.854362 31022 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:46.854435 31022 ts_tablet_manager.cc:1367] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:58:46.854380 27312 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.855021 31022 raft_consensus.cc:348] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.855119 31022 raft_consensus.cc:374] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.855146 31022 raft_consensus.cc:729] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.855252 31022 consensus_queue.cc:260] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.855508 31022 ts_tablet_manager.cc:1398] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:46.856065 31020 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:46.856194 31020 ts_tablet_manager.cc:1367] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:46.856992 24479 catalog_manager.cc:5484] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.857295 25032 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.857365 24897 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:46.859114 31020 raft_consensus.cc:348] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:46.859225 31020 raft_consensus.cc:374] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.859256 31020 raft_consensus.cc:729] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.859419 31020 consensus_queue.cc:260] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:46.863085 31020 ts_tablet_manager.cc:1398] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:58:46.863723 31020 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:46.865480 31020 tablet_bootstrap.cc:654] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:46.867190 31020 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:46.867326 31020 ts_tablet_manager.cc:1367] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:46.867702 31020 raft_consensus.cc:348] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.867815 31020 raft_consensus.cc:374] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:46.867864 31020 raft_consensus.cc:729] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:46.868477 31020 consensus_queue.cc:260] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.868822 24763 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720"
dest_uuid: "92cb785c165749eca49af5784756461c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50366
I20240502 01:58:46.868975 24763 raft_consensus.cc:595] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:46.869218 31020 ts_tablet_manager.cc:1398] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:46.872365 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:46.878756 30646 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:46.879101 30646 tablet.cc:1620] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:46.880352 30983 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:46.880623 30983 tablet.cc:1620] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:46.886173 30988 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:46.886467 30988 tablet.cc:1620] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:46.886796 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5091519dc99d4a0ea9ef6fe334effd0b] alter complete (version 2)
I20240502 01:58:46.986660 30591 raft_consensus.cc:482] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:46.986788 30591 raft_consensus.cc:504] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.987179 30591 leader_election.cc:288] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:46.987588 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e41964ae354b4d92385663ae940bc0" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:46.987588 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e41964ae354b4d92385663ae940bc0" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:46.987740 27445 raft_consensus.cc:2454] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:46.987740 24632 raft_consensus.cc:2454] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:46.988137 27514 leader_election.cc:302] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:46.988312 30591 raft_consensus.cc:2781] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:46.988364 30591 raft_consensus.cc:482] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:46.988404 30591 raft_consensus.cc:3037] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.989276 30591 raft_consensus.cc:504] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.989624 30591 leader_election.cc:288] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:46.989813 24480 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:46.990012 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e41964ae354b4d92385663ae940bc0" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:46.990123 24632 raft_consensus.cc:3037] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.990897 24632 raft_consensus.cc:2454] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:46.991809 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e41964ae354b4d92385663ae940bc0" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:46.991942 27445 raft_consensus.cc:3037] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:46.992856 27445 raft_consensus.cc:2454] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:46.993433 27513 leader_election.cc:302] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:46.993836 30591 raft_consensus.cc:2781] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:46.993952 30591 raft_consensus.cc:686] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:46.994082 30591 consensus_queue.cc:237] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:46.995896 30646 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:46.996109 24480 catalog_manager.cc:5484] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:46.996712 30983 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:46.999009 27445 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:46.998958 30988 tablet.cc:1722] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:47.000509 27017 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.003736 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=5091519dc99d4a0ea9ef6fe334effd0b] alter complete (version 3)
I20240502 01:58:47.045123 24897 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.045513 27017 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.048177 24763 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.049947 30398 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.098176 29098 raft_consensus.cc:482] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:47.098320 29098 raft_consensus.cc:504] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.098703 29098 leader_election.cc:288] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:47.099102 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97e5f78b03245cc86f70b1d50fcf723" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:47.099108 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97e5f78b03245cc86f70b1d50fcf723" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:47.099221 27312 raft_consensus.cc:2454] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:47.099244 27578 raft_consensus.cc:2454] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:47.099563 27110 leader_election.cc:302] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:47.099740 29098 raft_consensus.cc:2781] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:47.099790 29098 raft_consensus.cc:482] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:47.099819 29098 raft_consensus.cc:3037] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.100631 29098 raft_consensus.cc:504] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.100935 29098 leader_election.cc:288] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:47.101203 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97e5f78b03245cc86f70b1d50fcf723" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:47.101349 27312 raft_consensus.cc:3037] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.101341 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97e5f78b03245cc86f70b1d50fcf723" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:47.101481 27578 raft_consensus.cc:3037] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.102227 27578 raft_consensus.cc:2454] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:47.102335 27312 raft_consensus.cc:2454] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:47.102537 27113 leader_election.cc:302] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:47.102752 29098 raft_consensus.cc:2781] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:47.102828 29098 raft_consensus.cc:686] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:47.102955 29098 consensus_queue.cc:237] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.104619 24485 catalog_manager.cc:5484] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:47.106596 24483 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:47.106726 24483 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:47.108594 24897 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.109064 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 080e8c628aa948c0a4d5798936d089ee
I20240502 01:58:47.109586 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet 080e8c628aa948c0a4d5798936d089ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:47 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:47.109850 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 080e8c628aa948c0a4d5798936d089ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:47 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:47.110323 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 080e8c628aa948c0a4d5798936d089ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:47 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:47.111253 24897 raft_consensus.cc:1203] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:47.111053 31081 tablet_replica.cc:330] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:47.111745 31081 raft_consensus.cc:2229] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:47.112149 31081 raft_consensus.cc:2258] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.112900 31081 ts_tablet_manager.cc:1875] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:47.113135 30398 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.114334 31079 tablet_replica.cc:330] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:47.114503 31079 raft_consensus.cc:2229] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.114645 31079 raft_consensus.cc:2258] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.115387 31079 ts_tablet_manager.cc:1875] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:47.117244 31079 ts_tablet_manager.cc:1888] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:47.117311 31079 log.cc:1196] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/080e8c628aa948c0a4d5798936d089ee
I20240502 01:58:47.117558 31079 ts_tablet_manager.cc:1909] T 080e8c628aa948c0a4d5798936d089ee P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:47.118134 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table_new_name [id=5091519dc99d4a0ea9ef6fe334effd0b]) successfully deleted
I20240502 01:58:47.121829 31080 tablet_replica.cc:330] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:47.122013 31080 raft_consensus.cc:2229] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.122148 31080 raft_consensus.cc:2258] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.122426 31081 ts_tablet_manager.cc:1888] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:47.122638 31081 log.cc:1196] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/080e8c628aa948c0a4d5798936d089ee
I20240502 01:58:47.122731 31080 ts_tablet_manager.cc:1875] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:47.122912 31081 ts_tablet_manager.cc:1909] T 080e8c628aa948c0a4d5798936d089ee P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:47.123613 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table_new_name [id=5091519dc99d4a0ea9ef6fe334effd0b]) successfully deleted
I20240502 01:58:47.125890 31080 ts_tablet_manager.cc:1888] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:47.126024 31080 log.cc:1196] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/080e8c628aa948c0a4d5798936d089ee
I20240502 01:58:47.126331 31080 ts_tablet_manager.cc:1909] T 080e8c628aa948c0a4d5798936d089ee P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:47.126967 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 080e8c628aa948c0a4d5798936d089ee (table rebalancer_dynamic_table_new_name [id=5091519dc99d4a0ea9ef6fe334effd0b]) successfully deleted
I20240502 01:58:47.139077 30932 raft_consensus.cc:1050] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:47.139525 30932 consensus_queue.cc:237] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:47.141209 24763 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:47.141307 27177 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:47.141561 24897 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:47.141764 31093 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:47.141995 30932 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:47.143317 31099 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:47.145371 31099 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.146078 27177 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.146485 24763 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.148105 24897 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.148952 24483 catalog_manager.cc:5484] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:47.152446 24632 consensus_queue.cc:237] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:47.153326 24763 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720"
dest_uuid: "92cb785c165749eca49af5784756461c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50374
I20240502 01:58:47.153440 24763 raft_consensus.cc:595] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:47.153435 24897 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:47.155225 27177 raft_consensus.cc:1261] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:47.155352 31100 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.155678 31100 consensus_queue.cc:1035] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.156821 31100 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.157235 24897 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.158291 27177 raft_consensus.cc:2932] T 953a778a807a484b9e0376bdeeed8343 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.158793 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 953a778a807a484b9e0376bdeeed8343 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.159812 24480 catalog_manager.cc:5484] T 953a778a807a484b9e0376bdeeed8343 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 6, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.162732 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 953a778a807a484b9e0376bdeeed8343 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:47.162947 31080 tablet_replica.cc:330] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:47.163096 31080 raft_consensus.cc:2229] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.163228 31080 raft_consensus.cc:2258] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.163882 31080 ts_tablet_manager.cc:1875] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.164700 24897 consensus_queue.cc:237] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:47.165793 31080 ts_tablet_manager.cc:1888] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:47.165848 27177 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:47.165874 31080 log.cc:1196] T 953a778a807a484b9e0376bdeeed8343 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/953a778a807a484b9e0376bdeeed8343
I20240502 01:58:47.165906 27312 raft_consensus.cc:1261] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:47.166298 27093 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:47.166491 29574 consensus_queue.cc:1035] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:47.166996 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 953a778a807a484b9e0376bdeeed8343 (table rebalance_test_table_11 [id=1e630c9bf1cc46679cbb95ba9185ab23]) successfully deleted
I20240502 01:58:47.168200 29574 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.169503 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26b7501639064035a570b3b4c1cf083f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.170742 27312 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.173444 24485 catalog_manager.cc:5484] T 26b7501639064035a570b3b4c1cf083f P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index 8 to 10, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 2 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.173477 27177 raft_consensus.cc:2932] T 26b7501639064035a570b3b4c1cf083f P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.176443 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b7501639064035a570b3b4c1cf083f with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:47.176797 31120 tablet_replica.cc:330] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:47.177402 31120 raft_consensus.cc:2229] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.177596 31120 raft_consensus.cc:2258] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.178475 31120 ts_tablet_manager.cc:1875] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.180775 31120 ts_tablet_manager.cc:1888] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:47.180853 31120 log.cc:1196] T 26b7501639064035a570b3b4c1cf083f P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/26b7501639064035a570b3b4c1cf083f
I20240502 01:58:47.181491 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 26b7501639064035a570b3b4c1cf083f (table rebalance_test_table_5 [id=cb528248c7e84b92bf09f2953f8391ab]) successfully deleted
I20240502 01:58:47.204849 24763 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.205296 27017 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.207911 27177 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.208428 31069 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.211706 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:47.212366 31099 raft_consensus.cc:482] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:47.212476 31099 raft_consensus.cc:504] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:47.212864 31099 leader_election.cc:288] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:47.214303 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334184346780410a9f7efe298ae4d5e1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:47.214429 27576 raft_consensus.cc:2454] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:47.214761 24567 leader_election.cc:302] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:47.214962 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334184346780410a9f7efe298ae4d5e1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:47.215086 27312 raft_consensus.cc:2454] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:47.216907 31099 raft_consensus.cc:2781] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:47.216980 31099 raft_consensus.cc:482] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:47.217020 31099 raft_consensus.cc:3037] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.218252 31099 raft_consensus.cc:504] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:47.218624 31099 leader_election.cc:288] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:47.218884 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334184346780410a9f7efe298ae4d5e1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:47.219002 27312 raft_consensus.cc:3037] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.219907 25012 tablet_service.cc:1467] Processing CreateTablet for tablet b485f22f90f94595b629bc7da8a8e044 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a]), partition=
I20240502 01:58:47.220000 27312 raft_consensus.cc:2454] T 334184346780410a9f7efe298ae4d5e1 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:47.220151 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b485f22f90f94595b629bc7da8a8e044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.220315 24564 leader_election.cc:302] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:58:47.220467 31099 raft_consensus.cc:2781] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:47.220489 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334184346780410a9f7efe298ae4d5e1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:47.220547 31099 raft_consensus.cc:686] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:47.220602 27576 raft_consensus.cc:3037] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.220656 27424 tablet_service.cc:1467] Processing CreateTablet for tablet b485f22f90f94595b629bc7da8a8e044 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a]), partition=
I20240502 01:58:47.220664 31099 consensus_queue.cc:237] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:47.220870 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b485f22f90f94595b629bc7da8a8e044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.221568 27576 raft_consensus.cc:2454] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:47.222817 24479 catalog_manager.cc:5484] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:47.223060 30927 tablet_bootstrap.cc:492] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:47.224244 30927 tablet_bootstrap.cc:654] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.224902 31153 tablet_bootstrap.cc:492] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:47.225142 27289 tablet_service.cc:1467] Processing CreateTablet for tablet b485f22f90f94595b629bc7da8a8e044 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a]), partition=
I20240502 01:58:47.225363 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b485f22f90f94595b629bc7da8a8e044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.225883 30927 tablet_bootstrap.cc:492] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:47.225978 30927 ts_tablet_manager.cc:1367] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:47.226240 31153 tablet_bootstrap.cc:654] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.226398 30927 raft_consensus.cc:348] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.226500 30927 raft_consensus.cc:374] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.226542 30927 raft_consensus.cc:729] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.226651 30927 consensus_queue.cc:260] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.226893 30927 ts_tablet_manager.cc:1398] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:47.227797 30925 tablet_bootstrap.cc:492] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:47.228691 30925 tablet_bootstrap.cc:654] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.229177 31153 tablet_bootstrap.cc:492] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:47.229257 31153 ts_tablet_manager.cc:1367] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:47.229673 31153 raft_consensus.cc:348] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.229838 31153 raft_consensus.cc:374] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.229959 31153 raft_consensus.cc:729] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.230135 31153 consensus_queue.cc:260] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.230415 31153 ts_tablet_manager.cc:1398] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:47.231197 30925 tablet_bootstrap.cc:492] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:47.231277 30925 ts_tablet_manager.cc:1367] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:47.231647 30925 raft_consensus.cc:348] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.231745 30925 raft_consensus.cc:374] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.231774 30925 raft_consensus.cc:729] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.231889 30925 consensus_queue.cc:260] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.232136 30925 ts_tablet_manager.cc:1398] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:47.237809 24763 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720"
dest_uuid: "92cb785c165749eca49af5784756461c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50386
I20240502 01:58:47.237964 24763 raft_consensus.cc:595] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:47.242554 27177 raft_consensus.cc:1203] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:58:47.255326 30738 raft_consensus.cc:482] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:47.255434 30738 raft_consensus.cc:504] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.255874 30738 leader_election.cc:288] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:47.256135 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b485f22f90f94595b629bc7da8a8e044" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:47.256263 25032 raft_consensus.cc:2454] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:47.256273 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b485f22f90f94595b629bc7da8a8e044" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:47.256412 27445 raft_consensus.cc:2454] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:47.256520 27243 leader_election.cc:302] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:47.256706 30738 raft_consensus.cc:2781] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:47.256757 30738 raft_consensus.cc:482] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:47.256789 30738 raft_consensus.cc:3037] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.256788 28393 raft_consensus.cc:979] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: : Instructing follower f46bcd2bc124471981349b1e2726253b to start an election
I20240502 01:58:47.256908 28393 raft_consensus.cc:1067] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Signalling peer f46bcd2bc124471981349b1e2726253b to start an election
I20240502 01:58:47.257396 24897 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
from {username='slave'} at 127.17.243.2:46963
I20240502 01:58:47.257499 24897 raft_consensus.cc:482] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:47.257542 24897 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:47.257627 30738 raft_consensus.cc:504] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.257957 30738 leader_election.cc:288] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:47.258237 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b485f22f90f94595b629bc7da8a8e044" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:47.258347 25032 raft_consensus.cc:3037] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.258404 24897 raft_consensus.cc:504] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:47.258455 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b485f22f90f94595b629bc7da8a8e044" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:47.258545 27445 raft_consensus.cc:3037] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.258843 24897 leader_election.cc:288] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 2 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:47.259187 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:47.259258 25032 raft_consensus.cc:2454] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:47.259313 27312 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:47.259491 27445 raft_consensus.cc:2454] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:47.259691 27243 leader_election.cc:302] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:47.259954 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:47.260071 25032 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:47.260291 27312 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 2.
I20240502 01:58:47.260736 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92382869887b43f9a23a2c0ea0012720" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:47.260847 24763 raft_consensus.cc:3032] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:47.260916 24763 raft_consensus.cc:729] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:47.260973 25032 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 2.
I20240502 01:58:47.261088 24763 consensus_queue.cc:260] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:47.261176 30738 raft_consensus.cc:2781] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:47.261260 24830 leader_election.cc:302] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:47.261298 24763 raft_consensus.cc:3037] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:47.261337 30738 raft_consensus.cc:686] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:47.261432 29574 raft_consensus.cc:2781] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:47.261512 29574 raft_consensus.cc:686] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 2 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:47.261489 30738 consensus_queue.cc:237] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.261655 29574 consensus_queue.cc:237] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:47.262272 24763 raft_consensus.cc:2454] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 2.
I20240502 01:58:47.263598 24479 catalog_manager.cc:5484] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:47.263753 24485 catalog_manager.cc:5484] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 1 to 2, leader changed from 92cb785c165749eca49af5784756461c (127.17.243.2) to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 2 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:47.340196 31175 ts_tablet_manager.cc:902] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:47.340667 31175 tablet_copy_client.cc:308] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.5:40239
I20240502 01:58:47.341111 27182 tablet_copy_service.cc:140] P 9dd619a595e94004ac0f7f5a9054e7cc: Received BeginTabletCopySession request for tablet 8c28e5d1164e4dfb997f2d7fec77429e from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:55443)
I20240502 01:58:47.341212 27182 tablet_copy_service.cc:161] P 9dd619a595e94004ac0f7f5a9054e7cc: Beginning new tablet copy session on tablet 8c28e5d1164e4dfb997f2d7fec77429e from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:55443: session id = f46bcd2bc124471981349b1e2726253b-8c28e5d1164e4dfb997f2d7fec77429e
I20240502 01:58:47.342016 27182 tablet_copy_source_session.cc:205] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:47.342842 31175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e5d1164e4dfb997f2d7fec77429e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.344661 31175 tablet_copy_client.cc:783] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:47.344802 31175 tablet_copy_client.cc:647] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:47.345760 31175 tablet_copy_client.cc:523] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:47.346829 31175 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:47.349833 31175 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:47.350068 31175 tablet_bootstrap.cc:492] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:47.350216 31175 ts_tablet_manager.cc:1367] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.001s
I20240502 01:58:47.350590 31175 raft_consensus.cc:348] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:47.350695 31175 raft_consensus.cc:729] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:47.350782 31175 consensus_queue.cc:260] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:47.351032 31175 ts_tablet_manager.cc:1398] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:47.351485 27182 tablet_copy_service.cc:342] P 9dd619a595e94004ac0f7f5a9054e7cc: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-8c28e5d1164e4dfb997f2d7fec77429e received from {username='slave'} at 127.17.243.3:55443
I20240502 01:58:47.351552 27182 tablet_copy_service.cc:433] P 9dd619a595e94004ac0f7f5a9054e7cc: ending tablet copy session f46bcd2bc124471981349b1e2726253b-8c28e5d1164e4dfb997f2d7fec77429e on tablet 8c28e5d1164e4dfb997f2d7fec77429e with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:47.375830 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:47.379690 27445 raft_consensus.cc:1261] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:47.379690 25032 raft_consensus.cc:1261] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:47.380095 30738 consensus_queue.cc:1035] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.380265 31172 consensus_queue.cc:1035] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.383626 30988 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:47.383940 30988 tablet.cc:1620] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:47.384025 30646 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:47.384320 30646 tablet.cc:1620] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:47.385155 24479 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet b485f22f90f94595b629bc7da8a8e044 (table rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a]). Expected version 1 got 0
I20240502 01:58:47.387341 24480 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet b485f22f90f94595b629bc7da8a8e044 (table rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a]). Expected version 1 got 0
I20240502 01:58:47.387377 31185 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:47.388561 31185 tablet.cc:1620] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:47.390758 24485 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet b485f22f90f94595b629bc7da8a8e044 (table rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a]). Expected version 1 got 0
I20240502 01:58:47.390965 24480 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a] alter complete (version 1)
I20240502 01:58:47.493995 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:47.499080 30988 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:47.499418 30988 tablet.cc:1620] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:47.499577 31185 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:47.499847 31185 tablet.cc:1620] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:47.501879 30646 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:47.502210 30646 tablet.cc:1620] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:47.503780 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1dbc4a4b57264064b206a68a39e60d1a] alter complete (version 2)
I20240502 01:58:47.507138 30520 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.509871 30526 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.542258 30591 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.545156 31220 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.583791 25032 consensus_queue.cc:237] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:47.584883 27446 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:47.584980 24763 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:47.585331 30398 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.585529 31069 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.587182 30398 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:47.588636 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89580549c13d4c1ebdb2a5d24da2a01e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.588979 27446 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:47.588989 24763 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:47.590339 24483 catalog_manager.cc:5484] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 2 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:47.594998 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 89580549c13d4c1ebdb2a5d24da2a01e with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:47.595331 31223 tablet_replica.cc:330] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:47.595460 31223 raft_consensus.cc:2229] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.595566 31223 raft_consensus.cc:2258] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.596498 31223 ts_tablet_manager.cc:1875] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.598770 31223 ts_tablet_manager.cc:1888] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:47.598846 31223 log.cc:1196] T 89580549c13d4c1ebdb2a5d24da2a01e P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/89580549c13d4c1ebdb2a5d24da2a01e
I20240502 01:58:47.599339 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 89580549c13d4c1ebdb2a5d24da2a01e (table rebalance_test_table_9 [id=adcec1faee8a465983beaa8dad965891]) successfully deleted
I20240502 01:58:47.605907 24485 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:47.610440 30988 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:47.610795 30646 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:47.611027 31185 tablet.cc:1722] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:47.615005 24485 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=1dbc4a4b57264064b206a68a39e60d1a] alter complete (version 3)
I20240502 01:58:47.648425 25032 consensus_queue.cc:237] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:47.649524 30932 consensus_queue.cc:1035] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.649809 27177 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:47.649912 24763 raft_consensus.cc:1261] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:47.650243 27017 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.650435 31070 consensus_queue.cc:1035] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.652199 27017 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.652932 27177 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.653362 30932 consensus_queue.cc:1035] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:47.655860 24485 catalog_manager.cc:5484] T d136e006dd1048739156526af42fddd1 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 2 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.657621 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d136e006dd1048739156526af42fddd1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.658851 24763 raft_consensus.cc:2932] T d136e006dd1048739156526af42fddd1 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.660180 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet d136e006dd1048739156526af42fddd1 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:47.661691 31223 tablet_replica.cc:330] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:47.661852 31223 raft_consensus.cc:2229] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.662020 31223 raft_consensus.cc:2258] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.662648 31223 ts_tablet_manager.cc:1875] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.664346 31223 ts_tablet_manager.cc:1888] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:47.664445 31223 log.cc:1196] T d136e006dd1048739156526af42fddd1 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/d136e006dd1048739156526af42fddd1
I20240502 01:58:47.665167 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet d136e006dd1048739156526af42fddd1 (table rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]) successfully deleted
I20240502 01:58:47.666239 29098 raft_consensus.cc:1050] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:47.666630 29098 consensus_queue.cc:237] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:47.668020 27312 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:47.668115 24897 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:47.668246 27446 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:47.668462 31227 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:47.668658 30526 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:47.668849 30520 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:47.670719 29098 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.671188 24897 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.671200 27446 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.673753 24479 catalog_manager.cc:5484] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.674496 27312 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.677414 27177 consensus_queue.cc:237] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:47.679121 24897 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:47.679251 27446 raft_consensus.cc:1261] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:47.679508 29098 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:47.679689 30520 consensus_queue.cc:1035] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:47.681269 29098 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.682008 24897 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.682106 27446 raft_consensus.cc:2932] T 4f2436f6b4344a2294be04f84f52f5ff P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:47.683454 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f2436f6b4344a2294be04f84f52f5ff with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.684305 24483 catalog_manager.cc:5484] T 4f2436f6b4344a2294be04f84f52f5ff P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:47.687134 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2436f6b4344a2294be04f84f52f5ff with delete_type TABLET_DATA_TOMBSTONED (TS 1049b4c8783b48f39be22bd113ad349f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:47.687469 31235 tablet_replica.cc:330] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:47.687651 31235 raft_consensus.cc:2229] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.687796 31235 raft_consensus.cc:2258] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.688359 31235 ts_tablet_manager.cc:1875] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.690495 31235 ts_tablet_manager.cc:1888] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:47.690615 31235 log.cc:1196] T 4f2436f6b4344a2294be04f84f52f5ff P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/4f2436f6b4344a2294be04f84f52f5ff
I20240502 01:58:47.691104 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 4f2436f6b4344a2294be04f84f52f5ff (table rebalancer_extra_table_0 [id=d6cfb30521c74e7690021006b75a4282]) successfully deleted
I20240502 01:58:47.697167 25034 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.697630 27093 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.718432 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:47.718567 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:47.720592 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet b485f22f90f94595b629bc7da8a8e044
I20240502 01:58:47.721213 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet b485f22f90f94595b629bc7da8a8e044 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:47 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:47.721220 27289 tablet_service.cc:1514] Processing DeleteTablet for tablet b485f22f90f94595b629bc7da8a8e044 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:47 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:47.721304 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet b485f22f90f94595b629bc7da8a8e044 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:47 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:47.721443 31235 tablet_replica.cc:330] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:47.721580 31235 raft_consensus.cc:2229] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:47.721762 31268 tablet_replica.cc:330] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:47.721786 31235 raft_consensus.cc:2258] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.721897 31268 raft_consensus.cc:2229] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.722143 31267 tablet_replica.cc:330] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:47.722337 31267 raft_consensus.cc:2229] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.722544 31268 raft_consensus.cc:2258] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.722606 31235 ts_tablet_manager.cc:1875] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:47.722769 31267 raft_consensus.cc:2258] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.723616 31268 ts_tablet_manager.cc:1875] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:47.724318 31267 ts_tablet_manager.cc:1875] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:47.724743 31235 ts_tablet_manager.cc:1888] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:47.724804 31235 log.cc:1196] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/b485f22f90f94595b629bc7da8a8e044
I20240502 01:58:47.725044 31235 ts_tablet_manager.cc:1909] T b485f22f90f94595b629bc7da8a8e044 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:47.725662 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet b485f22f90f94595b629bc7da8a8e044 (table rebalancer_dynamic_table_new_name [id=1dbc4a4b57264064b206a68a39e60d1a]) successfully deleted
I20240502 01:58:47.725991 31268 ts_tablet_manager.cc:1888] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:47.726119 31268 log.cc:1196] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/b485f22f90f94595b629bc7da8a8e044
I20240502 01:58:47.726373 31268 ts_tablet_manager.cc:1909] T b485f22f90f94595b629bc7da8a8e044 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:47.726910 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet b485f22f90f94595b629bc7da8a8e044 (table rebalancer_dynamic_table_new_name [id=1dbc4a4b57264064b206a68a39e60d1a]) successfully deleted
I20240502 01:58:47.727015 31267 ts_tablet_manager.cc:1888] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:47.727092 31267 log.cc:1196] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/b485f22f90f94595b629bc7da8a8e044
I20240502 01:58:47.727294 31267 ts_tablet_manager.cc:1909] T b485f22f90f94595b629bc7da8a8e044 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:47.727860 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet b485f22f90f94595b629bc7da8a8e044 (table rebalancer_dynamic_table_new_name [id=1dbc4a4b57264064b206a68a39e60d1a]) successfully deleted
I20240502 01:58:47.730126 24897 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:47.731498 27446 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:47.731542 24763 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:58:47.731832 24832 consensus_peers.cc:485] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91d0d9525b5f4384861dc50d0bf4967d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:47.731923 31271 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.732124 27093 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.733963 29574 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:47.734280 24763 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:47.736812 24479 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:47.737288 27446 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:47.745325 24897 raft_consensus.cc:1203] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:47.755234 24632 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:47.756866 27446 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:47.757027 24897 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:58:47.757197 24566 consensus_peers.cc:485] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): Couldn't send request to peer 9dd619a595e94004ac0f7f5a9054e7cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69182e5f4ae7401f89d22f063fa8df63. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:47.757328 30932 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.757882 30932 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.759289 31093 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:47.759768 27446 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:47.760422 24897 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } } }
I20240502 01:58:47.762755 24480 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 6 to 7, NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:47.775552 30398 raft_consensus.cc:1050] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:47.775938 30398 consensus_queue.cc:237] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:47.777328 24897 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:47.777475 24632 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:47.777592 27177 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:47.777777 31069 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:47.777953 31070 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:47.778259 27017 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:47.780349 27017 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.780982 24897 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.782222 24632 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.783352 27177 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.785010 24479 catalog_manager.cc:5484] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 4 to 5, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.789058 25032 consensus_queue.cc:237] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:47.790160 24632 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:47.790256 27177 raft_consensus.cc:1261] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:47.790586 30398 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.790797 27017 consensus_queue.cc:1035] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.792080 24763 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.792512 31070 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.793982 31271 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.794688 24471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61ad1a07bc7a4fbcb901792d138e2843 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.795490 24632 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.795701 24480 catalog_manager.cc:5484] T 61ad1a07bc7a4fbcb901792d138e2843 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:47.796026 27177 raft_consensus.cc:2932] T 61ad1a07bc7a4fbcb901792d138e2843 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:47.797211 27312 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:47.797755 31271 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:47.799755 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ad1a07bc7a4fbcb901792d138e2843 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:47.800390 31223 tablet_replica.cc:330] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:47.800540 31223 raft_consensus.cc:2229] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.800698 31223 raft_consensus.cc:2258] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.801465 31223 ts_tablet_manager.cc:1875] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.803659 31223 ts_tablet_manager.cc:1888] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:47.803735 31223 log.cc:1196] T 61ad1a07bc7a4fbcb901792d138e2843 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/61ad1a07bc7a4fbcb901792d138e2843
I20240502 01:58:47.804035 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:47.804745 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 61ad1a07bc7a4fbcb901792d138e2843 (table rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]) successfully deleted
I20240502 01:58:47.809355 24897 consensus_queue.cc:237] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } }
I20240502 01:58:47.810655 27289 tablet_service.cc:1467] Processing CreateTablet for tablet 237661e1b1fa45e5a780c0af90e0138a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.810775 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 237661e1b1fa45e5a780c0af90e0138a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.810947 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237661e1b1fa45e5a780c0af90e0138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.811038 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237661e1b1fa45e5a780c0af90e0138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.811237 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 237661e1b1fa45e5a780c0af90e0138a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.811405 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 056c4148f1394bfab265dd06b10d88c1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.811476 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237661e1b1fa45e5a780c0af90e0138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.811661 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056c4148f1394bfab265dd06b10d88c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.813370 25032 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:47.813844 27424 tablet_service.cc:1467] Processing CreateTablet for tablet d0758e7e80294f8f97342db35da3761d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.814095 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0758e7e80294f8f97342db35da3761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.814396 27289 tablet_service.cc:1467] Processing CreateTablet for tablet d0758e7e80294f8f97342db35da3761d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.814503 31304 tablet_bootstrap.cc:492] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:47.814630 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0758e7e80294f8f97342db35da3761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.815043 27312 raft_consensus.cc:1261] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:47.815366 31304 tablet_bootstrap.cc:654] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.815500 27093 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.815855 27093 consensus_queue.cc:1035] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:47.816169 31305 tablet_bootstrap.cc:492] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:47.816366 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 056c4148f1394bfab265dd06b10d88c1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.816581 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056c4148f1394bfab265dd06b10d88c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.817119 31305 tablet_bootstrap.cc:654] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.817417 31302 tablet_bootstrap.cc:492] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:47.818368 31304 tablet_bootstrap.cc:492] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:47.818231 31271 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:47.818418 31302 tablet_bootstrap.cc:654] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.818445 31304 ts_tablet_manager.cc:1367] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:47.819064 31304 raft_consensus.cc:348] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.819188 31304 raft_consensus.cc:374] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.819232 31304 raft_consensus.cc:729] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.819460 31304 consensus_queue.cc:260] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.820266 27156 tablet_service.cc:1467] Processing CreateTablet for tablet d0758e7e80294f8f97342db35da3761d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.820427 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 056c4148f1394bfab265dd06b10d88c1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2b037ad0889f4085a5fe7706436c211a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:47.820551 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0758e7e80294f8f97342db35da3761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.820681 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056c4148f1394bfab265dd06b10d88c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.820909 24479 catalog_manager.cc:5484] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 5 to 7, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 2 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:47.821292 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92382869887b43f9a23a2c0ea0012720 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:47.822063 27312 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P 1049b4c8783b48f39be22bd113ad349f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:47.823268 24480 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:47.823619 31309 tablet_bootstrap.cc:492] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:47.824499 31309 tablet_bootstrap.cc:654] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.824930 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 92382869887b43f9a23a2c0ea0012720 with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:47.825508 31309 tablet_bootstrap.cc:492] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:47.825594 31309 ts_tablet_manager.cc:1367] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:47.826014 31309 raft_consensus.cc:348] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.826102 31309 raft_consensus.cc:374] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.826128 31309 raft_consensus.cc:729] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.826236 31309 consensus_queue.cc:260] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.826494 31309 ts_tablet_manager.cc:1398] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:47.826862 25032 raft_consensus.cc:2932] T 92382869887b43f9a23a2c0ea0012720 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: false } } }
I20240502 01:58:47.827982 31305 tablet_bootstrap.cc:492] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:47.828140 31305 ts_tablet_manager.cc:1367] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:58:47.828405 31303 tablet_bootstrap.cc:492] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:47.828768 31305 raft_consensus.cc:348] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:47.829100 31304 ts_tablet_manager.cc:1398] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 01:58:47.829192 31305 raft_consensus.cc:374] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.829278 31305 raft_consensus.cc:729] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.829998 31303 tablet_bootstrap.cc:654] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.831310 31302 tablet_bootstrap.cc:492] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:47.831462 31302 ts_tablet_manager.cc:1367] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:58:47.831789 31309 tablet_bootstrap.cc:492] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:47.832049 31302 raft_consensus.cc:348] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.832149 31302 raft_consensus.cc:374] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.832178 31302 raft_consensus.cc:729] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.832316 31302 consensus_queue.cc:260] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.833112 31302 ts_tablet_manager.cc:1398] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:47.833806 31309 tablet_bootstrap.cc:654] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.833336 31302 tablet_bootstrap.cc:492] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:47.834556 31311 tablet_replica.cc:330] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:47.834882 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 335bd097a4f74a3384fd9c46ea979b88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]), partition=
I20240502 01:58:47.834985 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 335bd097a4f74a3384fd9c46ea979b88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]), partition=
I20240502 01:58:47.835088 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335bd097a4f74a3384fd9c46ea979b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.835172 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335bd097a4f74a3384fd9c46ea979b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.835417 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 335bd097a4f74a3384fd9c46ea979b88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]), partition=
I20240502 01:58:47.835644 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 335bd097a4f74a3384fd9c46ea979b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:47.836428 31302 tablet_bootstrap.cc:654] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.837450 31305 consensus_queue.cc:260] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:47.837714 31302 tablet_bootstrap.cc:492] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:47.837985 31309 tablet_bootstrap.cc:492] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:47.838058 31309 ts_tablet_manager.cc:1367] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:58:47.838227 31302 ts_tablet_manager.cc:1367] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:47.838481 31309 raft_consensus.cc:348] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:47.838600 31309 raft_consensus.cc:374] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.838631 31309 raft_consensus.cc:729] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.838717 31302 raft_consensus.cc:348] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.838753 31309 consensus_queue.cc:260] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:47.838809 31302 raft_consensus.cc:374] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.838841 31302 raft_consensus.cc:729] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.838940 31302 consensus_queue.cc:260] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.839013 31309 ts_tablet_manager.cc:1398] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:47.839672 31305 ts_tablet_manager.cc:1398] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 01:58:47.840093 31311 raft_consensus.cc:2229] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.840261 31311 raft_consensus.cc:2258] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.840540 31305 tablet_bootstrap.cc:492] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:47.840795 31319 tablet_bootstrap.cc:492] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:47.841084 31311 ts_tablet_manager.cc:1875] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.841522 31305 tablet_bootstrap.cc:654] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.842121 31302 ts_tablet_manager.cc:1398] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:47.842777 31302 tablet_bootstrap.cc:492] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:47.843773 31302 tablet_bootstrap.cc:654] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.843900 31304 tablet_bootstrap.cc:492] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:47.844440 31319 tablet_bootstrap.cc:654] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.844727 31304 tablet_bootstrap.cc:654] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.845356 31311 ts_tablet_manager.cc:1888] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:47.845386 31302 tablet_bootstrap.cc:492] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:47.845466 31311 log.cc:1196] T 92382869887b43f9a23a2c0ea0012720 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/92382869887b43f9a23a2c0ea0012720
I20240502 01:58:47.845479 31302 ts_tablet_manager.cc:1367] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:47.845767 31304 tablet_bootstrap.cc:492] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:47.845852 31304 ts_tablet_manager.cc:1367] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:47.845867 31302 raft_consensus.cc:348] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:47.846024 31302 raft_consensus.cc:374] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.846058 31302 raft_consensus.cc:729] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.846103 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 92382869887b43f9a23a2c0ea0012720 (table rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]) successfully deleted
I20240502 01:58:47.846174 31302 consensus_queue.cc:260] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:47.846343 31304 raft_consensus.cc:348] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:47.846428 31302 ts_tablet_manager.cc:1398] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:47.846429 31304 raft_consensus.cc:374] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.846463 31304 raft_consensus.cc:729] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.846571 31304 consensus_queue.cc:260] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:47.846822 31304 ts_tablet_manager.cc:1398] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:47.846845 31303 tablet_bootstrap.cc:492] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:47.846935 31303 ts_tablet_manager.cc:1367] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240502 01:58:47.847303 31303 raft_consensus.cc:348] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.847379 31303 raft_consensus.cc:374] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.847397 31303 raft_consensus.cc:729] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.847481 31303 consensus_queue.cc:260] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.847726 31303 ts_tablet_manager.cc:1398] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:47.848996 31319 tablet_bootstrap.cc:492] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:47.849130 31319 ts_tablet_manager.cc:1367] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:47.849390 31305 tablet_bootstrap.cc:492] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:47.849468 31305 ts_tablet_manager.cc:1367] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:47.849684 31319 raft_consensus.cc:348] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:47.849831 31319 raft_consensus.cc:374] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.849881 31319 raft_consensus.cc:729] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.849829 31305 raft_consensus.cc:348] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.850006 31305 raft_consensus.cc:374] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.850080 31305 raft_consensus.cc:729] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.850113 31319 consensus_queue.cc:260] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:47.850224 31305 consensus_queue.cc:260] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.850479 31319 ts_tablet_manager.cc:1398] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:47.852128 31305 ts_tablet_manager.cc:1398] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:47.852358 31305 tablet_bootstrap.cc:492] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:47.853969 31305 tablet_bootstrap.cc:654] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:47.854990 31305 tablet_bootstrap.cc:492] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:47.855072 31305 ts_tablet_manager.cc:1367] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:47.855486 31305 raft_consensus.cc:348] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:47.855608 31305 raft_consensus.cc:374] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:47.855695 31305 raft_consensus.cc:729] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:47.855854 31305 consensus_queue.cc:260] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:47.856096 31305 ts_tablet_manager.cc:1398] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:47.950027 28447 raft_consensus.cc:482] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:47.950158 28447 raft_consensus.cc:504] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.950598 28447 leader_election.cc:288] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:47.950877 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0758e7e80294f8f97342db35da3761d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:47.951009 27177 raft_consensus.cc:2454] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:47.950996 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0758e7e80294f8f97342db35da3761d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:47.951121 27312 raft_consensus.cc:2454] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:47.951658 27379 leader_election.cc:302] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:47.951833 28447 raft_consensus.cc:2781] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:47.951912 28447 raft_consensus.cc:482] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:47.951964 28447 raft_consensus.cc:3037] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.952884 28447 raft_consensus.cc:504] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.953225 28447 leader_election.cc:288] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:47.953509 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0758e7e80294f8f97342db35da3761d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:47.953537 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0758e7e80294f8f97342db35da3761d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:47.953634 27177 raft_consensus.cc:3037] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.953639 27312 raft_consensus.cc:3037] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.954532 27312 raft_consensus.cc:2454] T d0758e7e80294f8f97342db35da3761d P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:47.954694 27177 raft_consensus.cc:2454] T d0758e7e80294f8f97342db35da3761d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:47.954778 27377 leader_election.cc:302] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:47.954943 28447 raft_consensus.cc:2781] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:47.955127 28447 raft_consensus.cc:686] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:47.955256 28447 consensus_queue.cc:237] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:47.957195 24483 catalog_manager.cc:5484] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:47.972512 30821 raft_consensus.cc:482] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:47.972626 30821 raft_consensus.cc:504] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.973021 30821 leader_election.cc:288] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:47.973364 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237661e1b1fa45e5a780c0af90e0138a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:47.973413 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237661e1b1fa45e5a780c0af90e0138a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:47.973490 27312 raft_consensus.cc:2454] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:47.973556 27576 raft_consensus.cc:2454] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:47.973753 24698 leader_election.cc:302] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:47.974011 30821 raft_consensus.cc:2781] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:47.974076 30821 raft_consensus.cc:482] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:47.974124 30821 raft_consensus.cc:3037] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.975001 30821 raft_consensus.cc:504] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.975343 30821 leader_election.cc:288] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:47.975593 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237661e1b1fa45e5a780c0af90e0138a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:47.975682 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237661e1b1fa45e5a780c0af90e0138a" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:47.975719 27312 raft_consensus.cc:3037] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.975806 27576 raft_consensus.cc:3037] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:47.976711 27576 raft_consensus.cc:2454] T 237661e1b1fa45e5a780c0af90e0138a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:47.976711 27312 raft_consensus.cc:2454] T 237661e1b1fa45e5a780c0af90e0138a P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:47.976994 24701 leader_election.cc:302] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:47.977164 30821 raft_consensus.cc:2781] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:47.977355 30821 raft_consensus.cc:686] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:47.977479 30821 consensus_queue.cc:237] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:47.979259 24483 catalog_manager.cc:5484] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.081343 31227 raft_consensus.cc:482] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:48.081456 31227 raft_consensus.cc:504] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:48.081826 31227 leader_election.cc:288] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:48.082178 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056c4148f1394bfab265dd06b10d88c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:48.082160 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056c4148f1394bfab265dd06b10d88c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:48.082304 27446 raft_consensus.cc:2454] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:48.082304 27312 raft_consensus.cc:2454] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:48.082583 27110 leader_election.cc:302] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:48.082762 31227 raft_consensus.cc:2781] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:48.082826 31227 raft_consensus.cc:482] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:48.082852 31227 raft_consensus.cc:3037] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.083931 31227 raft_consensus.cc:504] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:48.084281 31227 leader_election.cc:288] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:48.084551 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056c4148f1394bfab265dd06b10d88c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:48.084584 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056c4148f1394bfab265dd06b10d88c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:48.084657 27312 raft_consensus.cc:3037] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.084689 27446 raft_consensus.cc:3037] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.085417 27446 raft_consensus.cc:2454] T 056c4148f1394bfab265dd06b10d88c1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:48.085526 27312 raft_consensus.cc:2454] T 056c4148f1394bfab265dd06b10d88c1 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:48.085670 27113 leader_election.cc:302] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:48.085814 31227 raft_consensus.cc:2781] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:48.085892 31227 raft_consensus.cc:686] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:48.086053 31227 consensus_queue.cc:237] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:48.087704 24479 catalog_manager.cc:5484] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.167133 31069 raft_consensus.cc:482] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:48.167287 31069 raft_consensus.cc:504] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:48.167692 31069 leader_election.cc:288] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:48.167913 31351 ts_tablet_manager.cc:902] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:48.167997 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "335bd097a4f74a3384fd9c46ea979b88" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:48.168037 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "335bd097a4f74a3384fd9c46ea979b88" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:48.168133 27446 raft_consensus.cc:2454] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:48.168155 24763 raft_consensus.cc:2454] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:48.168376 31351 tablet_copy_client.cc:308] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:48.168447 24966 leader_election.cc:302] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:48.168591 31069 raft_consensus.cc:2781] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:48.168642 31069 raft_consensus.cc:482] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:48.168673 31069 raft_consensus.cc:3037] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.168802 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 91d0d9525b5f4384861dc50d0bf4967d from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:42087)
I20240502 01:58:48.168900 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 91d0d9525b5f4384861dc50d0bf4967d from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:42087: session id = 9dd619a595e94004ac0f7f5a9054e7cc-91d0d9525b5f4384861dc50d0bf4967d
I20240502 01:58:48.169822 24918 tablet_copy_source_session.cc:205] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.169790 31069 raft_consensus.cc:504] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:48.170132 31069 leader_election.cc:288] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:48.170465 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "335bd097a4f74a3384fd9c46ea979b88" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:48.170536 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "335bd097a4f74a3384fd9c46ea979b88" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:48.170595 24763 raft_consensus.cc:3037] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.170636 27446 raft_consensus.cc:3037] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.170809 31351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0d9525b5f4384861dc50d0bf4967d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.171399 24763 raft_consensus.cc:2454] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:48.171602 27446 raft_consensus.cc:2454] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:48.171643 24966 leader_election.cc:302] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:48.171794 31069 raft_consensus.cc:2781] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:48.171907 31069 raft_consensus.cc:686] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:48.172034 31069 consensus_queue.cc:237] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:48.173650 31351 tablet_copy_client.cc:783] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.173996 31351 tablet_copy_client.cc:647] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.175212 31351 tablet_copy_client.cc:523] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.175143 24479 catalog_manager.cc:5484] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:48.176095 31351 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:48.195384 31351 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.195634 31351 tablet_bootstrap.cc:492] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:48.195796 31351 ts_tablet_manager.cc:1367] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20240502 01:58:48.196159 31351 raft_consensus.cc:348] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:48.196278 31351 raft_consensus.cc:729] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:48.196379 31351 consensus_queue.cc:260] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:48.196655 31351 ts_tablet_manager.cc:1398] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:48.197136 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-91d0d9525b5f4384861dc50d0bf4967d received from {username='slave'} at 127.17.243.5:42087
I20240502 01:58:48.197224 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-91d0d9525b5f4384861dc50d0bf4967d on tablet 91d0d9525b5f4384861dc50d0bf4967d with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:48.219466 27290 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:48.220098 25012 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:58:48.221073 27155 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:48.227996 24744 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:58:48.231741 27557 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:48.237161 24610 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:48.243212 24877 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:58:48.246206 27424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:48.272728 31351 ts_tablet_manager.cc:902] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:48.273249 31351 tablet_copy_client.cc:308] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:48.273811 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 69182e5f4ae7401f89d22f063fa8df63 from peer 9dd619a595e94004ac0f7f5a9054e7cc ({username='slave'} at 127.17.243.5:52035)
I20240502 01:58:48.273957 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 69182e5f4ae7401f89d22f063fa8df63 from peer 9dd619a595e94004ac0f7f5a9054e7cc at {username='slave'} at 127.17.243.5:52035: session id = 9dd619a595e94004ac0f7f5a9054e7cc-69182e5f4ae7401f89d22f063fa8df63
I20240502 01:58:48.274967 24652 tablet_copy_source_session.cc:205] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.275933 31351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69182e5f4ae7401f89d22f063fa8df63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.282399 31351 tablet_copy_client.cc:783] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.282670 31351 tablet_copy_client.cc:647] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.283949 31351 tablet_copy_client.cc:523] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.285576 31351 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:48.301307 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:48.315399 24763 raft_consensus.cc:1261] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.315871 27445 raft_consensus.cc:1261] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.316041 31351 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.316316 31351 tablet_bootstrap.cc:492] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap complete.
I20240502 01:58:48.316490 31351 ts_tablet_manager.cc:1367] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.000s
I20240502 01:58:48.316869 31070 consensus_queue.cc:1035] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.316991 31351 raft_consensus.cc:348] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:48.317121 31351 raft_consensus.cc:729] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: LEARNER
I20240502 01:58:48.317152 31070 consensus_queue.cc:1035] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.317217 31351 consensus_queue.cc:260] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: NON_VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: true } }
I20240502 01:58:48.317610 31351 ts_tablet_manager.cc:1398] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:48.318449 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-69182e5f4ae7401f89d22f063fa8df63 received from {username='slave'} at 127.17.243.5:52035
I20240502 01:58:48.318537 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 9dd619a595e94004ac0f7f5a9054e7cc-69182e5f4ae7401f89d22f063fa8df63 on tablet 69182e5f4ae7401f89d22f063fa8df63 with peer 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:48.322753 24483 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]). Expected version 1 got 0
I20240502 01:58:48.324373 24479 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]). Expected version 1 got 0
I20240502 01:58:48.326089 30520 raft_consensus.cc:1050] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:48.326468 30520 consensus_queue.cc:237] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:48.328778 27312 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:48.328806 31391 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:48.329242 31391 tablet.cc:1620] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:48.330592 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:48.331693 24483 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]). Expected version 1 got 0
I20240502 01:58:48.333081 27576 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:48.333441 24897 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:48.333501 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:48.334051 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b] alter complete (version 1)
I20240502 01:58:48.335814 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:48.338068 31227 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.338426 24479 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b]). Expected version 1 got 0
I20240502 01:58:48.340101 27576 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.340526 31380 mvcc.cc:205] Tried to move back new op lower bound from 7023063565574615040 to 7023063564993998848. Current Snapshot: MvccSnapshot[applied={T|T < 7023063565574615040}]
I20240502 01:58:48.342100 31394 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:48.342510 31394 tablet.cc:1620] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:48.343382 24486 catalog_manager.cc:5484] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.344235 24897 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.345779 27312 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.354420 27177 consensus_queue.cc:237] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:48.355618 27576 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:48.356202 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:48.356215 24897 raft_consensus.cc:1261] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:48.356863 29098 consensus_queue.cc:1035] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:48.356953 31396 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:48.357591 31396 tablet.cc:1620] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:48.358424 31346 consensus_queue.cc:1035] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.359278 27177 consensus_queue.cc:237] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } }
W20240502 01:58:48.360822 27112 consensus_peers.cc:485] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): Couldn't send request to peer 588bfc090a5247598cd5d3cee5ad2007. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c8e4bca52cc47e89ad2b02d5035b1b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.361554 27578 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.362069 27445 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.362287 31389 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.362905 31389 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.364845 31389 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.365257 27576 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.368690 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c28e5d1164e4dfb997f2d7fec77429e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:48.368660 24481 catalog_manager.cc:5484] T 8c28e5d1164e4dfb997f2d7fec77429e P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.369279 24897 raft_consensus.cc:2932] T 8c28e5d1164e4dfb997f2d7fec77429e P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:48.371243 30520 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:48.373392 28447 consensus_queue.cc:1035] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.373972 27445 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:48.374722 27576 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:48.380260 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c28e5d1164e4dfb997f2d7fec77429e with delete_type TABLET_DATA_TOMBSTONED (TS 1049b4c8783b48f39be22bd113ad349f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:48.380748 24486 catalog_manager.cc:5484] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index -1 to 2, NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) added. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } } }
I20240502 01:58:48.385458 31408 tablet_replica.cc:330] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:48.385663 31408 raft_consensus.cc:2229] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:48.385875 31408 raft_consensus.cc:2258] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:48.386574 31408 ts_tablet_manager.cc:1875] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:48.388828 31408 ts_tablet_manager.cc:1888] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:48.388909 31408 log.cc:1196] T 8c28e5d1164e4dfb997f2d7fec77429e P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/8c28e5d1164e4dfb997f2d7fec77429e
I20240502 01:58:48.389079 24763 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:48.389446 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet 8c28e5d1164e4dfb997f2d7fec77429e (table rebalancer_extra_table_2 [id=3288464f517147278f1bde3e8e1966d0]) successfully deleted
I20240502 01:58:48.390498 27576 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:58:48.390749 24632 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 01:58:48.390792 24701 consensus_peers.cc:485] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44be529dec974eac8419efe9c19b30bf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.390991 30821 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:48.391630 30821 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:48.393002 31348 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.393468 27576 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.393738 24632 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.397215 24479 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 7 to 8, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 2 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.406213 24763 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:48.407503 27177 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:48.407532 25032 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:48.407934 31348 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:48.408229 31348 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:58:48.409768 24701 consensus_peers.cc:485] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63312ec15b034e47bc2dd484b57d4648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.410061 31348 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:48.410467 25032 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:48.412995 27177 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:48.413265 24481 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:48.428761 27576 consensus_queue.cc:237] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:48.430142 27445 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.430485 24632 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.430857 30591 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:58:48.431030 27510 consensus_peers.cc:485] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be -> Peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): Couldn't send request to peer e0b9e7df4e8543fa9b8e3ffc24330657. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2e41964ae354b4d92385663ae940bc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.431043 31220 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.432963 30591 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:48.434482 27445 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:48.435335 24632 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:48.438423 24486 catalog_manager.cc:5484] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.440898 31411 consensus_queue.cc:1035] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.447467 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:48.451344 31348 consensus_queue.cc:1035] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.454412 31391 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:48.454736 31391 tablet.cc:1620] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:48.456076 31394 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:48.456359 31394 tablet.cc:1620] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:48.460192 31396 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:48.460484 31396 tablet.cc:1620] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:48.463547 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=786d871a45274fb79d7db25f764f3c3b] alter complete (version 2)
I20240502 01:58:48.465770 27576 consensus_queue.cc:237] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
W20240502 01:58:48.467227 27514 consensus_peers.cc:485] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be -> Peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Couldn't send request to peer 1cc0a6338e0a467dbdc6efb3e67a6a09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abfa462d7c6d43f0ac967d298ee9050e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.467384 24632 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.467736 27177 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.468096 31220 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.468613 31220 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.470542 27867 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.471252 24630 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.472645 27177 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.473733 24479 catalog_manager.cc:5484] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 2, NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) added. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } } }
I20240502 01:58:48.490896 27445 consensus_queue.cc:237] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:48.492401 27312 raft_consensus.cc:1261] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.492604 27576 raft_consensus.cc:1261] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.492817 31346 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:58:48.493049 27380 consensus_peers.cc:485] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Couldn't send request to peer 92cb785c165749eca49af5784756461c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b2686f7b58341e9a0b1f2b27cfbb173. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.493168 28447 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.494848 28447 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.495465 27576 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.498704 24481 catalog_manager.cc:5484] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.500870 27312 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.510692 24763 consensus_queue.cc:237] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:48.512151 27576 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.512164 27177 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:48.512411 24698 consensus_peers.cc:485] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c -> Peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): Couldn't send request to peer e0b9e7df4e8543fa9b8e3ffc24330657. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b066a49a2b444d3961cac30b5cb6adf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.512578 30821 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.512754 31411 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.514624 31411 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:48.515302 27177 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:48.515909 27576 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } } }
I20240502 01:58:48.517905 24481 catalog_manager.cc:5484] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index -1 to 2, NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.527500 27177 consensus_queue.cc:237] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
W20240502 01:58:48.529142 27113 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8919df9a8a574882bad4995fc5df7e50. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.529441 27445 raft_consensus.cc:1261] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.529683 27312 raft_consensus.cc:1261] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.530117 29098 consensus_queue.cc:1035] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.530321 31389 consensus_queue.cc:1035] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.532181 31227 raft_consensus.cc:2932] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.532668 27312 raft_consensus.cc:2932] T 8919df9a8a574882bad4995fc5df7e50 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.532711 27445 raft_consensus.cc:2932] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.535341 24486 catalog_manager.cc:5484] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.543174 27177 consensus_queue.cc:237] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:48.544423 27312 raft_consensus.cc:1261] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:48.544709 27113 consensus_peers.cc:485] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Couldn't send request to peer 92cb785c165749eca49af5784756461c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c97e5f78b03245cc86f70b1d50fcf723. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.544751 29098 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.544974 27576 raft_consensus.cc:1261] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.545375 29098 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.546535 29098 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.546975 27312 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.547683 27576 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.549974 24481 catalog_manager.cc:5484] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.551499 31389 consensus_queue.cc:1035] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.556025 29098 consensus_queue.cc:1035] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.560705 27445 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:48.562005 27576 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:48.562093 25032 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:48.562466 31445 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:48.562659 31448 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:58:48.562857 27380 consensus_peers.cc:485] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Couldn't send request to peer 92cb785c165749eca49af5784756461c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adc992deb712458aa793eaa11019de1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.564421 31445 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.564858 25034 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.565613 27576 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.567852 24486 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 4 to 5, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:48.568014 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:48.573519 31391 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:48.573994 31396 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:48.575490 31394 tablet.cc:1722] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:48.581288 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=786d871a45274fb79d7db25f764f3c3b] alter complete (version 3)
I20240502 01:58:48.586796 25034 consensus_queue.cc:237] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:48.587937 24763 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:58:48.588048 27445 raft_consensus.cc:1261] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 01:58:48.588330 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89580549c13d4c1ebdb2a5d24da2a01e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.588423 31070 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:48.588600 31069 consensus_queue.cc:1035] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:48.590092 31070 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.592092 27445 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.592589 24479 catalog_manager.cc:5484] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 7 to 8, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 2 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.596896 24763 raft_consensus.cc:2932] T 89580549c13d4c1ebdb2a5d24da2a01e P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.616389 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:48.625483 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2cd42bb63448238ddfcca4cce87b74 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.625483 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2cd42bb63448238ddfcca4cce87b74 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.625597 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2cd42bb63448238ddfcca4cce87b74 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.625782 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2cd42bb63448238ddfcca4cce87b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.625787 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2cd42bb63448238ddfcca4cce87b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.625882 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2cd42bb63448238ddfcca4cce87b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.628041 31522 tablet_bootstrap.cc:492] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:48.628302 31523 tablet_bootstrap.cc:492] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:48.629274 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 45bfd9c929c14151b8ceeffa82d48555 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.629272 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 45bfd9c929c14151b8ceeffa82d48555 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.629387 27558 tablet_service.cc:1467] Processing CreateTablet for tablet b37610aa1329476188c0cd8b92973ca6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.629526 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bfd9c929c14151b8ceeffa82d48555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.629539 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bfd9c929c14151b8ceeffa82d48555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.629863 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37610aa1329476188c0cd8b92973ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.632117 27156 tablet_service.cc:1467] Processing CreateTablet for tablet b37610aa1329476188c0cd8b92973ca6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.632297 31526 tablet_bootstrap.cc:492] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:48.632390 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37610aa1329476188c0cd8b92973ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.632723 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 45bfd9c929c14151b8ceeffa82d48555 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.632908 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bfd9c929c14151b8ceeffa82d48555. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.631964 27290 tablet_service.cc:1467] Processing CreateTablet for tablet b37610aa1329476188c0cd8b92973ca6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=686d71cc5d9c4dc9bd705b2e51a49bea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:48.634974 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37610aa1329476188c0cd8b92973ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.635423 31525 tablet_bootstrap.cc:492] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:48.635962 31526 tablet_bootstrap.cc:654] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.636435 31525 tablet_bootstrap.cc:654] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.637087 31526 tablet_bootstrap.cc:492] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:48.637171 31526 ts_tablet_manager.cc:1367] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:48.638046 31526 raft_consensus.cc:348] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.638137 31526 raft_consensus.cc:374] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.638178 31526 raft_consensus.cc:729] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.638311 31526 consensus_queue.cc:260] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.638577 31522 tablet_bootstrap.cc:654] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.639129 31526 ts_tablet_manager.cc:1398] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:48.639298 31526 tablet_bootstrap.cc:492] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:48.639606 31523 tablet_bootstrap.cc:654] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.640249 31526 tablet_bootstrap.cc:654] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.640522 31524 tablet_bootstrap.cc:492] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:48.641505 31524 tablet_bootstrap.cc:654] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.641921 31523 tablet_bootstrap.cc:492] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:48.642019 31525 tablet_bootstrap.cc:492] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:48.642051 31523 ts_tablet_manager.cc:1367] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 01:58:48.642099 31525 ts_tablet_manager.cc:1367] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:48.642473 31523 raft_consensus.cc:348] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.642493 31525 raft_consensus.cc:348] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.642586 31523 raft_consensus.cc:374] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.642591 31525 raft_consensus.cc:374] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.642624 31525 raft_consensus.cc:729] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.642624 31523 raft_consensus.cc:729] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.642747 31525 consensus_queue.cc:260] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.642747 31523 consensus_queue.cc:260] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.643010 31523 ts_tablet_manager.cc:1398] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:48.643591 31523 tablet_bootstrap.cc:492] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:48.644574 31522 tablet_bootstrap.cc:492] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:48.644719 31522 ts_tablet_manager.cc:1367] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:58:48.644992 27177 raft_consensus.cc:1203] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:48.645264 31522 raft_consensus.cc:348] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.645434 31522 raft_consensus.cc:374] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.645517 31522 raft_consensus.cc:729] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.646411 31524 tablet_bootstrap.cc:492] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:48.646551 31524 ts_tablet_manager.cc:1367] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:48.646651 31522 consensus_queue.cc:260] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.647011 31522 ts_tablet_manager.cc:1398] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:48.647181 31524 raft_consensus.cc:348] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.648368 31525 ts_tablet_manager.cc:1398] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:48.648774 31523 tablet_bootstrap.cc:654] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.647460 31524 raft_consensus.cc:374] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.649295 31524 raft_consensus.cc:729] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.649471 31524 consensus_queue.cc:260] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.649768 31524 ts_tablet_manager.cc:1398] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:48.650820 31524 tablet_bootstrap.cc:492] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:48.651892 31524 tablet_bootstrap.cc:654] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.652823 31526 tablet_bootstrap.cc:492] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:48.652909 31526 ts_tablet_manager.cc:1367] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:58:48.653301 31526 raft_consensus.cc:348] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.653419 31526 raft_consensus.cc:374] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.653448 31526 raft_consensus.cc:729] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.653571 31526 consensus_queue.cc:260] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.653775 31524 tablet_bootstrap.cc:492] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:48.653836 31526 ts_tablet_manager.cc:1398] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:48.654059 31524 ts_tablet_manager.cc:1367] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:48.654465 31524 raft_consensus.cc:348] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.654569 31524 raft_consensus.cc:374] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.654608 31524 raft_consensus.cc:729] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.654721 31524 consensus_queue.cc:260] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.655257 31524 ts_tablet_manager.cc:1398] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:48.655442 31524 tablet_bootstrap.cc:492] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:48.655830 27177 raft_consensus.cc:1203] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:48.656337 31524 tablet_bootstrap.cc:654] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.657104 31523 tablet_bootstrap.cc:492] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:48.657433 31523 ts_tablet_manager.cc:1367] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240502 01:58:48.657507 31524 tablet_bootstrap.cc:492] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:48.657585 31524 ts_tablet_manager.cc:1367] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:48.657881 29098 raft_consensus.cc:482] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:48.657918 31523 raft_consensus.cc:348] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.657989 29098 raft_consensus.cc:504] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.658054 31523 raft_consensus.cc:374] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.658089 31523 raft_consensus.cc:729] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.657996 31524 raft_consensus.cc:348] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.658255 31524 raft_consensus.cc:374] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.658231 31523 consensus_queue.cc:260] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.658282 31524 raft_consensus.cc:729] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.658320 29098 leader_election.cc:288] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:48.658411 31524 consensus_queue.cc:260] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.658684 31524 ts_tablet_manager.cc:1398] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:48.659063 31523 ts_tablet_manager.cc:1398] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:48.659068 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2cd42bb63448238ddfcca4cce87b74" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:48.659178 24630 raft_consensus.cc:2454] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:48.659394 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2cd42bb63448238ddfcca4cce87b74" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:48.659505 27312 raft_consensus.cc:2454] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:48.659504 27112 leader_election.cc:302] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:48.659693 29098 raft_consensus.cc:2781] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:48.659739 29098 raft_consensus.cc:482] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:48.659771 29098 raft_consensus.cc:3037] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.660673 29098 raft_consensus.cc:504] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.661011 29098 leader_election.cc:288] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:48.661227 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2cd42bb63448238ddfcca4cce87b74" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:48.661232 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2cd42bb63448238ddfcca4cce87b74" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:48.661335 27312 raft_consensus.cc:3037] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.661335 24630 raft_consensus.cc:3037] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.662256 24630 raft_consensus.cc:2454] T 6d2cd42bb63448238ddfcca4cce87b74 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:48.662256 27312 raft_consensus.cc:2454] T 6d2cd42bb63448238ddfcca4cce87b74 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:48.662552 27112 leader_election.cc:302] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:48.662711 29098 raft_consensus.cc:2781] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:48.662815 29098 raft_consensus.cc:686] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:48.662941 29098 consensus_queue.cc:237] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:48.666184 24483 catalog_manager.cc:5484] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:48.684734 24483 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:48.684880 24483 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:48.687114 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 335bd097a4f74a3384fd9c46ea979b88
I20240502 01:58:48.687592 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 335bd097a4f74a3384fd9c46ea979b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:48 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:48.687721 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 335bd097a4f74a3384fd9c46ea979b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:48 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:48.687886 31574 tablet_replica.cc:330] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:48.688019 31575 tablet_replica.cc:330] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:48.688014 31574 raft_consensus.cc:2229] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:48.688001 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 335bd097a4f74a3384fd9c46ea979b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:48 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:48.688128 31574 raft_consensus.cc:2258] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:48.688143 31575 raft_consensus.cc:2229] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:48.688261 31575 raft_consensus.cc:2258] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:48.688638 31576 tablet_replica.cc:330] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:48.688791 31576 raft_consensus.cc:2229] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:48.688889 31575 ts_tablet_manager.cc:1875] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:48.689013 31576 raft_consensus.cc:2258] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:48.689414 31574 ts_tablet_manager.cc:1875] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:48.690690 31576 ts_tablet_manager.cc:1875] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:48.691737 31575 ts_tablet_manager.cc:1888] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:48.691867 31575 log.cc:1196] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/335bd097a4f74a3384fd9c46ea979b88
I20240502 01:58:48.692278 31575 ts_tablet_manager.cc:1909] T 335bd097a4f74a3384fd9c46ea979b88 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:48.692461 31574 ts_tablet_manager.cc:1888] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:48.692628 31574 log.cc:1196] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/335bd097a4f74a3384fd9c46ea979b88
I20240502 01:58:48.692955 31574 ts_tablet_manager.cc:1909] T 335bd097a4f74a3384fd9c46ea979b88 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:48.693033 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table_new_name [id=786d871a45274fb79d7db25f764f3c3b]) successfully deleted
I20240502 01:58:48.693686 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table_new_name [id=786d871a45274fb79d7db25f764f3c3b]) successfully deleted
I20240502 01:58:48.694589 31576 ts_tablet_manager.cc:1888] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:48.694653 31576 log.cc:1196] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/335bd097a4f74a3384fd9c46ea979b88
I20240502 01:58:48.694904 31576 ts_tablet_manager.cc:1909] T 335bd097a4f74a3384fd9c46ea979b88 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:48.695588 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 335bd097a4f74a3384fd9c46ea979b88 (table rebalancer_dynamic_table_new_name [id=786d871a45274fb79d7db25f764f3c3b]) successfully deleted
I20240502 01:58:48.711629 30738 raft_consensus.cc:482] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:48.711727 30738 raft_consensus.cc:504] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.712134 30738 leader_election.cc:288] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:48.712451 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:48.712599 27177 raft_consensus.cc:2454] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:48.712827 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:48.712963 27576 raft_consensus.cc:2454] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:48.713079 27245 leader_election.cc:302] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:48.713251 30738 raft_consensus.cc:2781] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:48.713307 30738 raft_consensus.cc:482] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:48.713341 30738 raft_consensus.cc:3037] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.714120 27576 consensus_queue.cc:237] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:48.714313 30738 raft_consensus.cc:504] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.714624 30738 leader_election.cc:288] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:48.714938 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:48.715059 27177 raft_consensus.cc:3037] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.715873 27445 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.716027 27177 raft_consensus.cc:2454] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:48.716042 24763 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:48.716084 27514 consensus_peers.cc:485] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60a07252321c4a0ebc16023c49015ffd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.716321 27245 leader_election.cc:302] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:48.716459 30738 raft_consensus.cc:2781] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:48.716406 31220 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.716598 30738 raft_consensus.cc:686] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:48.716707 30738 consensus_queue.cc:237] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:48.716745 31220 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.717788 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:48.717890 27576 raft_consensus.cc:3037] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.718616 24483 catalog_manager.cc:5484] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:48.718871 27576 raft_consensus.cc:2454] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:48.719095 27867 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.720284 27445 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.720774 24763 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.721782 24479 catalog_manager.cc:5484] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.730193 24763 consensus_queue.cc:237] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
W20240502 01:58:48.731653 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebe8db639839433d967583371351eacb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.731670 25034 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:48.732005 27576 raft_consensus.cc:1261] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:48.732057 31348 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:48.732426 31348 consensus_queue.cc:1035] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:48.733856 30821 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.734434 25034 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.734622 27576 raft_consensus.cc:2932] T ebe8db639839433d967583371351eacb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.737421 24481 catalog_manager.cc:5484] T ebe8db639839433d967583371351eacb P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:48.744625 31448 raft_consensus.cc:482] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:48.744735 31448 raft_consensus.cc:504] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.745124 31448 leader_election.cc:288] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:48.745493 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bfd9c929c14151b8ceeffa82d48555" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:48.745632 27177 raft_consensus.cc:2454] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:48.746006 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bfd9c929c14151b8ceeffa82d48555" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:48.746122 27576 raft_consensus.cc:2454] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:48.746376 27380 leader_election.cc:302] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:48.747108 31448 raft_consensus.cc:2781] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:48.747169 31448 raft_consensus.cc:482] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:48.747207 31448 raft_consensus.cc:3037] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.748136 31448 raft_consensus.cc:504] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.748448 31448 leader_election.cc:288] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:48.748703 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bfd9c929c14151b8ceeffa82d48555" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:48.748786 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bfd9c929c14151b8ceeffa82d48555" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:48.748824 27177 raft_consensus.cc:3037] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.748873 27576 raft_consensus.cc:3037] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.749686 27576 raft_consensus.cc:2454] T 45bfd9c929c14151b8ceeffa82d48555 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:48.749791 27177 raft_consensus.cc:2454] T 45bfd9c929c14151b8ceeffa82d48555 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:48.750001 27380 leader_election.cc:302] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:48.750146 31448 raft_consensus.cc:2781] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:48.750253 31448 raft_consensus.cc:686] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:48.750380 31448 consensus_queue.cc:237] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:48.752846 24479 catalog_manager.cc:5484] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.766935 27312 consensus_queue.cc:237] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:48.768357 27177 raft_consensus.cc:1261] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.768363 27576 raft_consensus.cc:1261] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.768729 30738 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:58:48.768921 27246 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84ae33b0741c48c090dda6d4a819ef02. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:48.769101 30738 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:48.770679 31620 raft_consensus.cc:2932] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.771037 27177 raft_consensus.cc:2932] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.771517 27576 raft_consensus.cc:2932] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:48.773705 24479 catalog_manager.cc:5484] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.788686 31638 ts_tablet_manager.cc:902] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: Initiating tablet copy from peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:48.789191 31638 tablet_copy_client.cc:308] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.5:40239
I20240502 01:58:48.789182 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:48.789716 27182 tablet_copy_service.cc:140] P 9dd619a595e94004ac0f7f5a9054e7cc: Received BeginTabletCopySession request for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 from peer 588bfc090a5247598cd5d3cee5ad2007 ({username='slave'} at 127.17.243.1:38227)
I20240502 01:58:48.789803 27182 tablet_copy_service.cc:161] P 9dd619a595e94004ac0f7f5a9054e7cc: Beginning new tablet copy session on tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 from peer 588bfc090a5247598cd5d3cee5ad2007 at {username='slave'} at 127.17.243.1:38227: session id = 588bfc090a5247598cd5d3cee5ad2007-3c8e4bca52cc47e89ad2b02d5035b1b7
I20240502 01:58:48.792021 27182 tablet_copy_source_session.cc:205] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.793174 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.795758 27290 tablet_service.cc:1467] Processing CreateTablet for tablet df151ac162924af9aa78db5e4c5f9363 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]), partition=
I20240502 01:58:48.795996 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df151ac162924af9aa78db5e4c5f9363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.796008 24877 tablet_service.cc:1467] Processing CreateTablet for tablet df151ac162924af9aa78db5e4c5f9363 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]), partition=
I20240502 01:58:48.796097 24744 tablet_service.cc:1467] Processing CreateTablet for tablet df151ac162924af9aa78db5e4c5f9363 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]), partition=
I20240502 01:58:48.796205 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df151ac162924af9aa78db5e4c5f9363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.796310 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df151ac162924af9aa78db5e4c5f9363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.798030 31523 tablet_bootstrap.cc:492] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:48.799000 31523 tablet_bootstrap.cc:654] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.799038 31648 tablet_bootstrap.cc:492] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:48.799376 31649 tablet_bootstrap.cc:492] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:48.799748 31638 tablet_copy_client.cc:783] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.799849 31648 tablet_bootstrap.cc:654] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.799983 31638 tablet_copy_client.cc:647] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.800295 31649 tablet_bootstrap.cc:654] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:48.801617 31523 tablet_bootstrap.cc:492] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:48.801704 31523 ts_tablet_manager.cc:1367] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:48.801836 31649 tablet_bootstrap.cc:492] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:48.801970 31649 ts_tablet_manager.cc:1367] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:48.802134 31523 raft_consensus.cc:348] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.802220 31523 raft_consensus.cc:374] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.802250 31523 raft_consensus.cc:729] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.802371 31523 consensus_queue.cc:260] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.802655 31523 ts_tablet_manager.cc:1398] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:48.802649 31649 raft_consensus.cc:348] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.802743 31649 raft_consensus.cc:374] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.802793 31649 raft_consensus.cc:729] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.801187 31638 tablet_copy_client.cc:523] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.802903 31649 consensus_queue.cc:260] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.803193 31649 ts_tablet_manager.cc:1398] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:48.804306 31638 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:48.807823 31638 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.807940 31648 tablet_bootstrap.cc:492] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:48.808022 31648 ts_tablet_manager.cc:1367] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:58:48.808046 31638 tablet_bootstrap.cc:492] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap complete.
I20240502 01:58:48.808188 31638 ts_tablet_manager.cc:1367] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240502 01:58:48.808626 31638 raft_consensus.cc:348] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } }
I20240502 01:58:48.808734 31638 raft_consensus.cc:729] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: LEARNER
I20240502 01:58:48.808825 31638 consensus_queue.cc:260] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: NON_VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: true } }
I20240502 01:58:48.809080 31638 ts_tablet_manager.cc:1398] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:48.809581 27182 tablet_copy_service.cc:342] P 9dd619a595e94004ac0f7f5a9054e7cc: Request end of tablet copy session 588bfc090a5247598cd5d3cee5ad2007-3c8e4bca52cc47e89ad2b02d5035b1b7 received from {username='slave'} at 127.17.243.1:38227
I20240502 01:58:48.809672 27182 tablet_copy_service.cc:433] P 9dd619a595e94004ac0f7f5a9054e7cc: ending tablet copy session 588bfc090a5247598cd5d3cee5ad2007-3c8e4bca52cc47e89ad2b02d5035b1b7 on tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 with peer 588bfc090a5247598cd5d3cee5ad2007
I20240502 01:58:48.809689 31648 raft_consensus.cc:348] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.809800 31648 raft_consensus.cc:374] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:48.809839 31648 raft_consensus.cc:729] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:48.809150 31654 ts_tablet_manager.cc:902] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: Initiating tablet copy from peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:48.809994 31648 consensus_queue.cc:260] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.810639 31654 tablet_copy_client.cc:308] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.8:44673
I20240502 01:58:48.810847 31648 ts_tablet_manager.cc:1398] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:48.814227 27599 tablet_copy_service.cc:140] P bb653087e8554c7591cf547dd969f3be: Received BeginTabletCopySession request for tablet b2e41964ae354b4d92385663ae940bc0 from peer e0b9e7df4e8543fa9b8e3ffc24330657 ({username='slave'} at 127.17.243.4:47413)
I20240502 01:58:48.814337 27599 tablet_copy_service.cc:161] P bb653087e8554c7591cf547dd969f3be: Beginning new tablet copy session on tablet b2e41964ae354b4d92385663ae940bc0 from peer e0b9e7df4e8543fa9b8e3ffc24330657 at {username='slave'} at 127.17.243.4:47413: session id = e0b9e7df4e8543fa9b8e3ffc24330657-b2e41964ae354b4d92385663ae940bc0
I20240502 01:58:48.815310 27599 tablet_copy_source_session.cc:205] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.815984 31598 raft_consensus.cc:482] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:48.816084 31598 raft_consensus.cc:504] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.816196 31654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e41964ae354b4d92385663ae940bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.816452 31598 leader_election.cc:288] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:48.816866 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df151ac162924af9aa78db5e4c5f9363" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:48.816995 24763 raft_consensus.cc:2454] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:48.817340 27246 leader_election.cc:302] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:48.817781 31598 raft_consensus.cc:2781] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:48.817842 31598 raft_consensus.cc:482] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:48.817875 31598 raft_consensus.cc:3037] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.818866 31598 raft_consensus.cc:504] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.818933 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df151ac162924af9aa78db5e4c5f9363" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:48.819023 24897 raft_consensus.cc:2454] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 0.
I20240502 01:58:48.819139 31598 leader_election.cc:288] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:48.819249 31654 tablet_copy_client.cc:783] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.819420 31654 tablet_copy_client.cc:647] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.819690 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df151ac162924af9aa78db5e4c5f9363" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:48.819794 24897 raft_consensus.cc:3037] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.819808 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df151ac162924af9aa78db5e4c5f9363" candidate_uuid: "1049b4c8783b48f39be22bd113ad349f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:48.819907 24763 raft_consensus.cc:3037] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:48.820592 31654 tablet_copy_client.cc:523] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.820685 24897 raft_consensus.cc:2454] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:48.820771 24763 raft_consensus.cc:2454] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1049b4c8783b48f39be22bd113ad349f in term 1.
I20240502 01:58:48.820981 27246 leader_election.cc:302] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:48.821151 31598 raft_consensus.cc:2781] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:48.821223 31598 raft_consensus.cc:686] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Becoming Leader. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Running, Role: LEADER
I20240502 01:58:48.821343 31598 consensus_queue.cc:237] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:48.821836 31654 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:48.823225 24486 catalog_manager.cc:5484] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1049b4c8783b48f39be22bd113ad349f (127.17.243.6). New cstate: current_term: 1 leader_uuid: "1049b4c8783b48f39be22bd113ad349f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:48.825152 31654 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.825389 31654 tablet_bootstrap.cc:492] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap complete.
I20240502 01:58:48.825551 31654 ts_tablet_manager.cc:1367] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:48.826045 31654 raft_consensus.cc:348] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:48.826153 31654 raft_consensus.cc:729] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: LEARNER
I20240502 01:58:48.826243 31654 consensus_queue.cc:260] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:48.826524 31654 ts_tablet_manager.cc:1398] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:48.827243 27599 tablet_copy_service.cc:342] P bb653087e8554c7591cf547dd969f3be: Request end of tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-b2e41964ae354b4d92385663ae940bc0 received from {username='slave'} at 127.17.243.4:47413
I20240502 01:58:48.827309 27599 tablet_copy_service.cc:433] P bb653087e8554c7591cf547dd969f3be: ending tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-b2e41964ae354b4d92385663ae940bc0 on tablet b2e41964ae354b4d92385663ae940bc0 with peer e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:58:48.908651 31721 ts_tablet_manager.cc:902] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:48.909447 31721 tablet_copy_client.cc:308] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:48.909879 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 63312ec15b034e47bc2dd484b57d4648 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:39965)
I20240502 01:58:48.910008 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 63312ec15b034e47bc2dd484b57d4648 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:39965: session id = bb653087e8554c7591cf547dd969f3be-63312ec15b034e47bc2dd484b57d4648
I20240502 01:58:48.910780 24785 tablet_copy_source_session.cc:205] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.912533 31721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63312ec15b034e47bc2dd484b57d4648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.914804 31721 tablet_copy_client.cc:783] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.915081 31721 tablet_copy_client.cc:647] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.916141 31721 tablet_copy_client.cc:523] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.917376 31721 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:48.932132 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:48.936388 24897 raft_consensus.cc:1261] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.936388 24763 raft_consensus.cc:1261] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 1049b4c8783b48f39be22bd113ad349f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:48.936807 31598 consensus_queue.cc:1035] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.936992 31620 consensus_queue.cc:1035] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:48.939713 24483 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]). Expected version 1 got 0
I20240502 01:58:48.941080 31396 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:48.941383 31396 tablet.cc:1620] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:48.941663 31721 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.941912 31721 tablet_bootstrap.cc:492] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:48.942092 31721 ts_tablet_manager.cc:1367] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.009s
I20240502 01:58:48.942241 24479 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]). Expected version 1 got 0
I20240502 01:58:48.942591 24481 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]). Expected version 1 got 0
I20240502 01:58:48.942536 31721 raft_consensus.cc:348] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:48.942656 31721 raft_consensus.cc:729] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:48.942746 31721 consensus_queue.cc:260] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:48.943105 31721 ts_tablet_manager.cc:1398] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:48.943161 24486 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]). Expected version 1 got 0
I20240502 01:58:48.943570 24785 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-63312ec15b034e47bc2dd484b57d4648 received from {username='slave'} at 127.17.243.8:39965
I20240502 01:58:48.943639 24785 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session bb653087e8554c7591cf547dd969f3be-63312ec15b034e47bc2dd484b57d4648 on tablet 63312ec15b034e47bc2dd484b57d4648 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:48.944448 31730 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:48.944794 31730 tablet.cc:1620] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:48.945386 24483 catalog_manager.cc:5576] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707) does not have the latest schema for tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]). Expected version 1 got 0
I20240502 01:58:48.946198 31731 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:48.946707 31731 tablet.cc:1620] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:48.947659 24483 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da]). Expected version 1 got 0
I20240502 01:58:48.949273 31732 ts_tablet_manager.cc:902] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: Initiating tablet copy from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:48.949788 31732 tablet_copy_client.cc:308] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.7:42021
I20240502 01:58:48.950675 27466 tablet_copy_service.cc:140] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Received BeginTabletCopySession request for tablet 9b2686f7b58341e9a0b1f2b27cfbb173 from peer 92cb785c165749eca49af5784756461c ({username='slave'} at 127.17.243.2:33651)
I20240502 01:58:48.950780 27466 tablet_copy_service.cc:161] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Beginning new tablet copy session on tablet 9b2686f7b58341e9a0b1f2b27cfbb173 from peer 92cb785c165749eca49af5784756461c at {username='slave'} at 127.17.243.2:33651: session id = 92cb785c165749eca49af5784756461c-9b2686f7b58341e9a0b1f2b27cfbb173
I20240502 01:58:48.951723 27466 tablet_copy_source_session.cc:205] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.952517 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da] alter complete (version 1)
I20240502 01:58:48.956077 31738 ts_tablet_manager.cc:902] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:48.956440 31732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2686f7b58341e9a0b1f2b27cfbb173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.956650 31738 tablet_copy_client.cc:308] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.8:44673
I20240502 01:58:48.957455 27599 tablet_copy_service.cc:140] P bb653087e8554c7591cf547dd969f3be: Received BeginTabletCopySession request for tablet abfa462d7c6d43f0ac967d298ee9050e from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:36915)
I20240502 01:58:48.957556 27599 tablet_copy_service.cc:161] P bb653087e8554c7591cf547dd969f3be: Beginning new tablet copy session on tablet abfa462d7c6d43f0ac967d298ee9050e from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:36915: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-abfa462d7c6d43f0ac967d298ee9050e
I20240502 01:58:48.958302 27599 tablet_copy_source_session.cc:205] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.959013 31738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfa462d7c6d43f0ac967d298ee9050e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.959193 31732 tablet_copy_client.cc:783] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.959347 31732 tablet_copy_client.cc:647] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.960685 31732 tablet_copy_client.cc:523] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.960969 31738 tablet_copy_client.cc:783] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.961174 31738 tablet_copy_client.cc:647] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:48.961802 31732 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:48.962211 31738 tablet_copy_client.cc:523] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:48.963419 31738 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:48.965111 31732 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.965303 31732 tablet_bootstrap.cc:492] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: Bootstrap complete.
I20240502 01:58:48.965435 31732 ts_tablet_manager.cc:1367] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:48.966230 31732 raft_consensus.cc:348] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:48.966419 31732 raft_consensus.cc:729] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: LEARNER
I20240502 01:58:48.966588 31732 consensus_queue.cc:260] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:48.966948 31738 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:48.967144 31738 tablet_bootstrap.cc:492] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:48.967288 31738 ts_tablet_manager.cc:1367] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:48.967670 31732 ts_tablet_manager.cc:1398] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:48.968086 27466 tablet_copy_service.cc:342] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Request end of tablet copy session 92cb785c165749eca49af5784756461c-9b2686f7b58341e9a0b1f2b27cfbb173 received from {username='slave'} at 127.17.243.2:33651
I20240502 01:58:48.968154 27466 tablet_copy_service.cc:433] P 1cc0a6338e0a467dbdc6efb3e67a6a09: ending tablet copy session 92cb785c165749eca49af5784756461c-9b2686f7b58341e9a0b1f2b27cfbb173 on tablet 9b2686f7b58341e9a0b1f2b27cfbb173 with peer 92cb785c165749eca49af5784756461c
I20240502 01:58:48.968295 31738 raft_consensus.cc:348] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:48.968446 31738 raft_consensus.cc:729] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:48.968560 31738 consensus_queue.cc:260] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:48.968988 31738 ts_tablet_manager.cc:1398] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:48.969482 27599 tablet_copy_service.cc:342] P bb653087e8554c7591cf547dd969f3be: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-abfa462d7c6d43f0ac967d298ee9050e received from {username='slave'} at 127.17.243.7:36915
I20240502 01:58:48.969552 27599 tablet_copy_service.cc:433] P bb653087e8554c7591cf547dd969f3be: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-abfa462d7c6d43f0ac967d298ee9050e on tablet abfa462d7c6d43f0ac967d298ee9050e with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:48.995276 31654 ts_tablet_manager.cc:902] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:48.995720 31654 tablet_copy_client.cc:308] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:48.996150 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 9b066a49a2b444d3961cac30b5cb6adf from peer e0b9e7df4e8543fa9b8e3ffc24330657 ({username='slave'} at 127.17.243.4:41349)
I20240502 01:58:48.996238 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 9b066a49a2b444d3961cac30b5cb6adf from peer e0b9e7df4e8543fa9b8e3ffc24330657 at {username='slave'} at 127.17.243.4:41349: session id = e0b9e7df4e8543fa9b8e3ffc24330657-9b066a49a2b444d3961cac30b5cb6adf
I20240502 01:58:48.996902 24785 tablet_copy_source_session.cc:205] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:48.997690 31654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b066a49a2b444d3961cac30b5cb6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:48.999619 31654 tablet_copy_client.cc:783] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:48.999768 31654 tablet_copy_client.cc:647] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.000838 31654 tablet_copy_client.cc:523] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.001842 31654 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:49.004766 31738 ts_tablet_manager.cc:902] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:49.004854 31654 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.005158 31654 tablet_bootstrap.cc:492] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap complete.
I20240502 01:58:49.005250 31738 tablet_copy_client.cc:308] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:49.005338 31654 ts_tablet_manager.cc:1367] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:49.005681 24785 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet 44be529dec974eac8419efe9c19b30bf from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 ({username='slave'} at 127.17.243.7:54897)
I20240502 01:58:49.005765 24785 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet 44be529dec974eac8419efe9c19b30bf from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 at {username='slave'} at 127.17.243.7:54897: session id = 1cc0a6338e0a467dbdc6efb3e67a6a09-44be529dec974eac8419efe9c19b30bf
I20240502 01:58:49.005754 31654 raft_consensus.cc:348] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:49.005875 31654 raft_consensus.cc:729] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: LEARNER
I20240502 01:58:49.006083 31654 consensus_queue.cc:260] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: NON_VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: true } }
I20240502 01:58:49.006387 31654 ts_tablet_manager.cc:1398] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:49.006665 24785 tablet_copy_source_session.cc:205] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.006911 24784 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-9b066a49a2b444d3961cac30b5cb6adf received from {username='slave'} at 127.17.243.4:41349
I20240502 01:58:49.006968 24784 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session e0b9e7df4e8543fa9b8e3ffc24330657-9b066a49a2b444d3961cac30b5cb6adf on tablet 9b066a49a2b444d3961cac30b5cb6adf with peer e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:58:49.007437 31738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44be529dec974eac8419efe9c19b30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.009322 31738 tablet_copy_client.cc:783] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.009469 31738 tablet_copy_client.cc:647] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.010481 31738 tablet_copy_client.cc:523] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.011395 31738 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:49.030259 31738 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.030470 31738 tablet_bootstrap.cc:492] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap complete.
I20240502 01:58:49.030596 31738 ts_tablet_manager.cc:1367] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20240502 01:58:49.030932 31738 raft_consensus.cc:348] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:49.031033 31738 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: LEARNER
I20240502 01:58:49.031116 31738 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: NON_VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: true } }
I20240502 01:58:49.031376 31738 ts_tablet_manager.cc:1398] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:49.031805 24784 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-44be529dec974eac8419efe9c19b30bf received from {username='slave'} at 127.17.243.7:54897
I20240502 01:58:49.031878 24784 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 1cc0a6338e0a467dbdc6efb3e67a6a09-44be529dec974eac8419efe9c19b30bf on tablet 44be529dec974eac8419efe9c19b30bf with peer 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:49.047523 31732 ts_tablet_manager.cc:902] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: Initiating tablet copy from peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:49.047968 31732 tablet_copy_client.cc:308] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.7:42021
I20240502 01:58:49.048381 27466 tablet_copy_service.cc:140] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Received BeginTabletCopySession request for tablet adc992deb712458aa793eaa11019de1f from peer 92cb785c165749eca49af5784756461c ({username='slave'} at 127.17.243.2:33651)
I20240502 01:58:49.048480 27466 tablet_copy_service.cc:161] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Beginning new tablet copy session on tablet adc992deb712458aa793eaa11019de1f from peer 92cb785c165749eca49af5784756461c at {username='slave'} at 127.17.243.2:33651: session id = 92cb785c165749eca49af5784756461c-adc992deb712458aa793eaa11019de1f
I20240502 01:58:49.049185 27466 tablet_copy_source_session.cc:205] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.050016 31732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc992deb712458aa793eaa11019de1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.051692 31732 tablet_copy_client.cc:783] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.051842 31732 tablet_copy_client.cc:647] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.052853 31732 tablet_copy_client.cc:523] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.053880 31732 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:49.056897 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:49.059553 31732 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.059809 31732 tablet_bootstrap.cc:492] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: Bootstrap complete.
I20240502 01:58:49.059968 31732 ts_tablet_manager.cc:1367] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:58:49.060461 31732 raft_consensus.cc:348] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:49.060593 31732 raft_consensus.cc:729] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: LEARNER
I20240502 01:58:49.060700 31732 consensus_queue.cc:260] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:49.060992 31732 ts_tablet_manager.cc:1398] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:49.062330 27466 tablet_copy_service.cc:342] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Request end of tablet copy session 92cb785c165749eca49af5784756461c-adc992deb712458aa793eaa11019de1f received from {username='slave'} at 127.17.243.2:33651
I20240502 01:58:49.062301 31730 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:49.062431 27466 tablet_copy_service.cc:433] P 1cc0a6338e0a467dbdc6efb3e67a6a09: ending tablet copy session 92cb785c165749eca49af5784756461c-adc992deb712458aa793eaa11019de1f on tablet adc992deb712458aa793eaa11019de1f with peer 92cb785c165749eca49af5784756461c
I20240502 01:58:49.062628 31730 tablet.cc:1620] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: MemRowSet was empty: no flush needed.
I20240502 01:58:49.063041 31731 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:49.063339 31731 tablet.cc:1620] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:49.065168 31396 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:49.065450 31396 tablet.cc:1620] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:49.069159 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=525e9bc7b57942eeb740db5a17fa49da] alter complete (version 2)
I20240502 01:58:49.083353 31544 raft_consensus.cc:1050] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:49.083802 31544 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:49.085418 24897 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.085417 27177 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.085431 27446 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.085858 31767 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.086135 31768 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.086493 31544 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.090286 30520 consensus_queue.cc:1035] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.090711 31544 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.091434 27176 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.093199 27446 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.094573 24897 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.096336 31227 consensus_queue.cc:1035] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.097402 24483 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:49.098920 31781 ts_tablet_manager.cc:902] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:49.102111 31781 tablet_copy_client.cc:308] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.4:42879
I20240502 01:58:49.102567 25054 tablet_copy_service.cc:140] P e0b9e7df4e8543fa9b8e3ffc24330657: Received BeginTabletCopySession request for tablet 89580549c13d4c1ebdb2a5d24da2a01e from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:43875)
I20240502 01:58:49.102656 25054 tablet_copy_service.cc:161] P e0b9e7df4e8543fa9b8e3ffc24330657: Beginning new tablet copy session on tablet 89580549c13d4c1ebdb2a5d24da2a01e from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:43875: session id = 1049b4c8783b48f39be22bd113ad349f-89580549c13d4c1ebdb2a5d24da2a01e
I20240502 01:58:49.103535 25054 tablet_copy_source_session.cc:205] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.103508 24630 consensus_queue.cc:237] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:49.104467 31781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89580549c13d4c1ebdb2a5d24da2a01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.104848 27176 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:49.104892 27446 raft_consensus.cc:1261] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:49.105619 31100 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:49.107057 31781 tablet_copy_client.cc:783] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.107244 31781 tablet_copy_client.cc:647] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.107452 31768 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.107826 27446 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.109115 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 69182e5f4ae7401f89d22f063fa8df63 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.110929 24486 catalog_manager.cc:5484] T 69182e5f4ae7401f89d22f063fa8df63 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 8 to 9, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.111261 31100 consensus_queue.cc:1035] T 69182e5f4ae7401f89d22f063fa8df63 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:49.111366 31781 tablet_copy_client.cc:523] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.112776 31781 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:49.113523 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 69182e5f4ae7401f89d22f063fa8df63 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:49.113600 27176 raft_consensus.cc:2932] T 69182e5f4ae7401f89d22f063fa8df63 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.116171 31788 tablet_replica.cc:330] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:49.116340 31788 raft_consensus.cc:2229] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.116593 31788 raft_consensus.cc:2258] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.117779 31788 ts_tablet_manager.cc:1875] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.118161 31789 ts_tablet_manager.cc:902] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:49.118695 31789 tablet_copy_client.cc:308] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.8:44673
I20240502 01:58:49.119228 27599 tablet_copy_service.cc:140] P bb653087e8554c7591cf547dd969f3be: Received BeginTabletCopySession request for tablet 60a07252321c4a0ebc16023c49015ffd from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:43389)
I20240502 01:58:49.119349 27599 tablet_copy_service.cc:161] P bb653087e8554c7591cf547dd969f3be: Beginning new tablet copy session on tablet 60a07252321c4a0ebc16023c49015ffd from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:43389: session id = f46bcd2bc124471981349b1e2726253b-60a07252321c4a0ebc16023c49015ffd
I20240502 01:58:49.120373 27599 tablet_copy_source_session.cc:205] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.121284 31789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a07252321c4a0ebc16023c49015ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.123108 31732 ts_tablet_manager.cc:902] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: Initiating tablet copy from peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:49.123611 31732 tablet_copy_client.cc:308] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.5:40239
I20240502 01:58:49.124132 27182 tablet_copy_service.cc:140] P 9dd619a595e94004ac0f7f5a9054e7cc: Received BeginTabletCopySession request for tablet c97e5f78b03245cc86f70b1d50fcf723 from peer 92cb785c165749eca49af5784756461c ({username='slave'} at 127.17.243.2:58133)
I20240502 01:58:49.124225 27182 tablet_copy_service.cc:161] P 9dd619a595e94004ac0f7f5a9054e7cc: Beginning new tablet copy session on tablet c97e5f78b03245cc86f70b1d50fcf723 from peer 92cb785c165749eca49af5784756461c at {username='slave'} at 127.17.243.2:58133: session id = 92cb785c165749eca49af5784756461c-c97e5f78b03245cc86f70b1d50fcf723
I20240502 01:58:49.124477 31789 tablet_copy_client.cc:783] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.125099 27182 tablet_copy_source_session.cc:205] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.125522 31789 tablet_copy_client.cc:647] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.126349 31788 ts_tablet_manager.cc:1888] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:58:49.126432 31788 log.cc:1196] T 69182e5f4ae7401f89d22f063fa8df63 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/69182e5f4ae7401f89d22f063fa8df63
I20240502 01:58:49.127076 31732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97e5f78b03245cc86f70b1d50fcf723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.127190 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 69182e5f4ae7401f89d22f063fa8df63 (table rebalance_test_table_7 [id=274c56ba08dd4ee9818c08dfe5c19560]) successfully deleted
I20240502 01:58:49.129110 31732 tablet_copy_client.cc:783] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.129261 31732 tablet_copy_client.cc:647] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.130244 31789 tablet_copy_client.cc:523] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.131569 31789 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:49.134615 31732 tablet_copy_client.cc:523] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.136003 31732 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:49.140188 31789 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.140827 31732 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.141249 31732 tablet_bootstrap.cc:492] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: Bootstrap complete.
I20240502 01:58:49.141436 31732 ts_tablet_manager.cc:1367] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:49.141986 31732 raft_consensus.cc:348] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:49.142145 31732 raft_consensus.cc:729] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: LEARNER
I20240502 01:58:49.142362 31732 consensus_queue.cc:260] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:49.142722 31732 ts_tablet_manager.cc:1398] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:49.143887 27182 tablet_copy_service.cc:342] P 9dd619a595e94004ac0f7f5a9054e7cc: Request end of tablet copy session 92cb785c165749eca49af5784756461c-c97e5f78b03245cc86f70b1d50fcf723 received from {username='slave'} at 127.17.243.2:58133
I20240502 01:58:49.143962 27182 tablet_copy_service.cc:433] P 9dd619a595e94004ac0f7f5a9054e7cc: ending tablet copy session 92cb785c165749eca49af5784756461c-c97e5f78b03245cc86f70b1d50fcf723 on tablet c97e5f78b03245cc86f70b1d50fcf723 with peer 92cb785c165749eca49af5784756461c
I20240502 01:58:49.144408 31789 tablet_bootstrap.cc:492] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:49.144583 31789 ts_tablet_manager.cc:1367] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240502 01:58:49.145364 29574 raft_consensus.cc:1050] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER 9dd619a595e94004ac0f7f5a9054e7cc to VOTER
I20240502 01:58:49.145764 31781 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.145737 31598 consensus_queue.cc:1035] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.145748 29574 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:49.146061 31781 tablet_bootstrap.cc:492] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:49.145954 31789 raft_consensus.cc:348] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:49.146113 31789 raft_consensus.cc:729] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:49.146214 31781 ts_tablet_manager.cc:1367] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240502 01:58:49.146224 31789 consensus_queue.cc:260] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:49.147140 31781 raft_consensus.cc:348] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:49.147531 31781 raft_consensus.cc:729] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:49.147682 27176 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.147682 31781 consensus_queue.cc:260] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:49.146783 31789 ts_tablet_manager.cc:1398] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:49.148121 31781 ts_tablet_manager.cc:1398] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:49.148803 27446 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.149148 31805 ts_tablet_manager.cc:902] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:49.149132 31271 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.149742 25054 tablet_copy_service.cc:342] P e0b9e7df4e8543fa9b8e3ffc24330657: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-89580549c13d4c1ebdb2a5d24da2a01e received from {username='slave'} at 127.17.243.6:43875
I20240502 01:58:49.149816 25054 tablet_copy_service.cc:433] P e0b9e7df4e8543fa9b8e3ffc24330657: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-89580549c13d4c1ebdb2a5d24da2a01e on tablet 89580549c13d4c1ebdb2a5d24da2a01e with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:49.150460 27599 tablet_copy_service.cc:342] P bb653087e8554c7591cf547dd969f3be: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-60a07252321c4a0ebc16023c49015ffd received from {username='slave'} at 127.17.243.3:43389
I20240502 01:58:49.150542 27599 tablet_copy_service.cc:433] P bb653087e8554c7591cf547dd969f3be: ending tablet copy session f46bcd2bc124471981349b1e2726253b-60a07252321c4a0ebc16023c49015ffd on tablet 60a07252321c4a0ebc16023c49015ffd with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:49.151021 31805 tablet_copy_client.cc:308] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.5:40239
I20240502 01:58:49.151316 31271 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.151613 27182 tablet_copy_service.cc:140] P 9dd619a595e94004ac0f7f5a9054e7cc: Received BeginTabletCopySession request for tablet 8919df9a8a574882bad4995fc5df7e50 from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:55443)
I20240502 01:58:49.151715 27182 tablet_copy_service.cc:161] P 9dd619a595e94004ac0f7f5a9054e7cc: Beginning new tablet copy session on tablet 8919df9a8a574882bad4995fc5df7e50 from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:55443: session id = f46bcd2bc124471981349b1e2726253b-8919df9a8a574882bad4995fc5df7e50
I20240502 01:58:49.152344 31620 consensus_queue.cc:1035] T b37610aa1329476188c0cd8b92973ca6 P 1049b4c8783b48f39be22bd113ad349f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.152662 27182 tablet_copy_source_session.cc:205] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.155014 24763 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.156533 31805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8919df9a8a574882bad4995fc5df7e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.158023 31814 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.160238 31805 tablet_copy_client.cc:783] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.160538 31805 tablet_copy_client.cc:647] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.161573 31805 tablet_copy_client.cc:523] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.162153 31813 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.162798 31805 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:49.164650 24763 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.164672 27446 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.167655 24486 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.169549 27176 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:49.171109 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:49.172266 31805 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.173350 31805 tablet_bootstrap.cc:492] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:49.173511 31805 ts_tablet_manager.cc:1367] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:58:49.173986 31805 raft_consensus.cc:348] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:49.174100 31805 raft_consensus.cc:729] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:49.174199 31805 consensus_queue.cc:260] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:49.174471 31805 ts_tablet_manager.cc:1398] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:49.174979 27182 tablet_copy_service.cc:342] P 9dd619a595e94004ac0f7f5a9054e7cc: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-8919df9a8a574882bad4995fc5df7e50 received from {username='slave'} at 127.17.243.3:55443
I20240502 01:58:49.175050 27182 tablet_copy_service.cc:433] P 9dd619a595e94004ac0f7f5a9054e7cc: ending tablet copy session f46bcd2bc124471981349b1e2726253b-8919df9a8a574882bad4995fc5df7e50 on tablet 8919df9a8a574882bad4995fc5df7e50 with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:49.178290 31730 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:49.179075 31731 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:49.183498 31396 tablet.cc:1722] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:49.184279 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=525e9bc7b57942eeb740db5a17fa49da] alter complete (version 3)
I20240502 01:58:49.186547 24630 raft_consensus.cc:1203] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.230221 31805 ts_tablet_manager.cc:902] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:49.230696 31805 tablet_copy_client.cc:308] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.6:41707
I20240502 01:58:49.231182 27332 tablet_copy_service.cc:140] P 1049b4c8783b48f39be22bd113ad349f: Received BeginTabletCopySession request for tablet 84ae33b0741c48c090dda6d4a819ef02 from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:45317)
I20240502 01:58:49.231289 27332 tablet_copy_service.cc:161] P 1049b4c8783b48f39be22bd113ad349f: Beginning new tablet copy session on tablet 84ae33b0741c48c090dda6d4a819ef02 from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:45317: session id = f46bcd2bc124471981349b1e2726253b-84ae33b0741c48c090dda6d4a819ef02
I20240502 01:58:49.232218 27332 tablet_copy_source_session.cc:205] T 84ae33b0741c48c090dda6d4a819ef02 P 1049b4c8783b48f39be22bd113ad349f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.233373 31805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ae33b0741c48c090dda6d4a819ef02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.235652 31805 tablet_copy_client.cc:783] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.235793 31805 tablet_copy_client.cc:647] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.237673 31805 tablet_copy_client.cc:523] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.238735 31805 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:49.242101 31805 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.242322 31805 tablet_bootstrap.cc:492] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:49.242452 31805 ts_tablet_manager.cc:1367] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:49.242777 31805 raft_consensus.cc:348] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:49.242859 31805 raft_consensus.cc:729] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:49.242956 31805 consensus_queue.cc:260] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:49.243324 31805 ts_tablet_manager.cc:1398] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:49.243747 27332 tablet_copy_service.cc:342] P 1049b4c8783b48f39be22bd113ad349f: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-84ae33b0741c48c090dda6d4a819ef02 received from {username='slave'} at 127.17.243.3:45317
I20240502 01:58:49.243819 27332 tablet_copy_service.cc:433] P 1049b4c8783b48f39be22bd113ad349f: ending tablet copy session f46bcd2bc124471981349b1e2726253b-84ae33b0741c48c090dda6d4a819ef02 on tablet 84ae33b0741c48c090dda6d4a819ef02 with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:49.252593 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49974
I20240502 01:58:49.252694 24897 raft_consensus.cc:595] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:49.261173 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:49.266232 25034 consensus_queue.cc:237] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:49.267283 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad9731874143179216c33d478aa6f4 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.267583 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad9731874143179216c33d478aa6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.267694 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 0dba43199ac941dcaefe4f0236e90156 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.267971 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dba43199ac941dcaefe4f0236e90156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.268383 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 5084f1b5d4314a5fa71e117128f0c737 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.268455 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 0dba43199ac941dcaefe4f0236e90156 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.268636 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5084f1b5d4314a5fa71e117128f0c737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.268736 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dba43199ac941dcaefe4f0236e90156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.270157 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad9731874143179216c33d478aa6f4 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.270280 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 5084f1b5d4314a5fa71e117128f0c737 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.270366 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad9731874143179216c33d478aa6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.270545 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5084f1b5d4314a5fa71e117128f0c737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.270941 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 5084f1b5d4314a5fa71e117128f0c737 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.271173 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5084f1b5d4314a5fa71e117128f0c737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.271942 31648 tablet_bootstrap.cc:492] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:49.272656 31648 tablet_bootstrap.cc:654] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.272740 31908 tablet_bootstrap.cc:492] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:49.272933 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad9731874143179216c33d478aa6f4 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.273154 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad9731874143179216c33d478aa6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.273635 31908 tablet_bootstrap.cc:654] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.274098 31906 tablet_bootstrap.cc:492] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:49.274725 31908 tablet_bootstrap.cc:492] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:49.274801 31908 ts_tablet_manager.cc:1367] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:49.274946 31906 tablet_bootstrap.cc:654] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.275220 31908 raft_consensus.cc:348] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.275347 31908 raft_consensus.cc:374] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.275453 31908 raft_consensus.cc:729] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.275455 24897 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:49.275454 27446 raft_consensus.cc:1261] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:49.275671 31908 consensus_queue.cc:260] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.275933 31648 tablet_bootstrap.cc:492] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:49.276014 31648 ts_tablet_manager.cc:1367] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
W20240502 01:58:49.276113 24963 consensus_peers.cc:485] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b8667a49f1a44339122040b0122be0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.276356 31906 tablet_bootstrap.cc:492] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:49.276433 31906 ts_tablet_manager.cc:1367] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:49.276422 31648 raft_consensus.cc:348] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.276510 31648 raft_consensus.cc:374] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.276540 31648 raft_consensus.cc:729] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.276644 31648 consensus_queue.cc:260] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.276805 31491 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:49.276912 31648 ts_tablet_manager.cc:1398] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:49.276933 31908 ts_tablet_manager.cc:1398] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:49.277477 31906 raft_consensus.cc:348] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.277611 31906 raft_consensus.cc:374] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.277664 31906 raft_consensus.cc:729] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.277690 31907 tablet_bootstrap.cc:492] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:49.277794 31906 consensus_queue.cc:260] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.278084 31069 consensus_queue.cc:1035] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:49.278812 31906 ts_tablet_manager.cc:1398] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:49.279004 31906 tablet_bootstrap.cc:492] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:49.280058 31906 tablet_bootstrap.cc:654] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.280455 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 0dba43199ac941dcaefe4f0236e90156 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3bec5ed6d22b43ba9db2d2e2658709c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:49.280695 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dba43199ac941dcaefe4f0236e90156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.282748 31523 tablet_bootstrap.cc:492] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:49.283700 31523 tablet_bootstrap.cc:654] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.286881 31905 tablet_bootstrap.cc:492] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:49.288031 31905 tablet_bootstrap.cc:654] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.289281 31523 tablet_bootstrap.cc:492] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f: No bootstrap required, opened a new log
I20240502 01:58:49.289366 31523 ts_tablet_manager.cc:1367] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:58:49.289799 31523 raft_consensus.cc:348] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.289896 31523 raft_consensus.cc:374] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.289963 31523 raft_consensus.cc:729] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.290083 31523 consensus_queue.cc:260] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.290118 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:49.290217 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:49.290375 31523 ts_tablet_manager.cc:1398] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:49.290649 31905 tablet_bootstrap.cc:492] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:49.290781 31905 ts_tablet_manager.cc:1367] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:49.291441 31905 raft_consensus.cc:348] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.292637 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet df151ac162924af9aa78db5e4c5f9363
I20240502 01:58:49.292695 31905 raft_consensus.cc:374] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.292789 31905 raft_consensus.cc:729] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.292958 31905 consensus_queue.cc:260] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.293253 31905 ts_tablet_manager.cc:1398] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:49.293254 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet df151ac162924af9aa78db5e4c5f9363 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:49 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:49.294056 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet df151ac162924af9aa78db5e4c5f9363 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:49 UTC) from {username='slave'} at 127.0.0.1:45938
I20240502 01:58:49.295836 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet df151ac162924af9aa78db5e4c5f9363 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:49 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:49.295572 31918 tablet_replica.cc:330] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:49.295930 31906 tablet_bootstrap.cc:492] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:49.295998 31906 ts_tablet_manager.cc:1367] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240502 01:58:49.296010 31788 tablet_replica.cc:330] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:49.296015 31918 raft_consensus.cc:2229] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.296128 31788 raft_consensus.cc:2229] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.296134 31918 raft_consensus.cc:2258] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.296227 31788 raft_consensus.cc:2258] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.296833 31918 ts_tablet_manager.cc:1875] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:49.296739 31069 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:49.297999 31788 ts_tablet_manager.cc:1875] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:49.298979 27446 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:49.299284 31918 ts_tablet_manager.cc:1888] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:49.299355 31918 log.cc:1196] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/df151ac162924af9aa78db5e4c5f9363
I20240502 01:58:49.299667 31918 ts_tablet_manager.cc:1909] T df151ac162924af9aa78db5e4c5f9363 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:49.300261 31788 ts_tablet_manager.cc:1888] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:49.300341 31788 log.cc:1196] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/df151ac162924af9aa78db5e4c5f9363
I20240502 01:58:49.300345 31906 raft_consensus.cc:348] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.300432 31906 raft_consensus.cc:374] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.300452 31906 raft_consensus.cc:729] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.300550 31906 consensus_queue.cc:260] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.300598 31788 ts_tablet_manager.cc:1909] T df151ac162924af9aa78db5e4c5f9363 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:49.300812 31906 ts_tablet_manager.cc:1398] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:49.301676 24482 catalog_manager.cc:5484] T 9b8667a49f1a44339122040b0122be0d P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:49.302350 31781 ts_tablet_manager.cc:902] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:49.302784 31781 tablet_copy_client.cc:308] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.2:45237
I20240502 01:58:49.303181 24784 tablet_copy_service.cc:140] P 92cb785c165749eca49af5784756461c: Received BeginTabletCopySession request for tablet ebe8db639839433d967583371351eacb from peer 1049b4c8783b48f39be22bd113ad349f ({username='slave'} at 127.17.243.6:59521)
I20240502 01:58:49.303270 24784 tablet_copy_service.cc:161] P 92cb785c165749eca49af5784756461c: Beginning new tablet copy session on tablet ebe8db639839433d967583371351eacb from peer 1049b4c8783b48f39be22bd113ad349f at {username='slave'} at 127.17.243.6:59521: session id = 1049b4c8783b48f39be22bd113ad349f-ebe8db639839433d967583371351eacb
I20240502 01:58:49.303988 24784 tablet_copy_source_session.cc:205] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:49.304701 31781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8db639839433d967583371351eacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.304730 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table_new_name [id=525e9bc7b57942eeb740db5a17fa49da]) successfully deleted
I20240502 01:58:49.305042 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table_new_name [id=525e9bc7b57942eeb740db5a17fa49da]) successfully deleted
I20240502 01:58:49.306823 31907 tablet_bootstrap.cc:654] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.307601 31919 tablet_replica.cc:330] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: stopping tablet replica
I20240502 01:58:49.307790 31919 raft_consensus.cc:2229] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:49.307790 31781 tablet_copy_client.cc:783] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:49.308095 31781 tablet_copy_client.cc:647] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:49.308137 31919 raft_consensus.cc:2258] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.309269 31907 tablet_bootstrap.cc:492] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:49.309420 31907 ts_tablet_manager.cc:1367] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240502 01:58:49.310040 31907 raft_consensus.cc:348] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.310153 31907 raft_consensus.cc:374] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.310181 31907 raft_consensus.cc:729] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.310345 31907 consensus_queue.cc:260] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.310727 31907 ts_tablet_manager.cc:1398] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:49.311041 31907 tablet_bootstrap.cc:492] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:49.312075 24897 raft_consensus.cc:2932] T 9b8667a49f1a44339122040b0122be0d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } } }
I20240502 01:58:49.312801 31907 tablet_bootstrap.cc:654] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.316211 31781 tablet_copy_client.cc:523] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:49.317363 31781 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: Bootstrap starting.
I20240502 01:58:49.317863 25034 raft_consensus.cc:1203] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.322624 31919 ts_tablet_manager.cc:1875] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:49.324589 31919 ts_tablet_manager.cc:1888] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:49.324651 31919 log.cc:1196] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-5/wal/wals/df151ac162924af9aa78db5e4c5f9363
I20240502 01:58:49.324865 31919 ts_tablet_manager.cc:1909] T df151ac162924af9aa78db5e4c5f9363 P 1049b4c8783b48f39be22bd113ad349f: Deleting consensus metadata
I20240502 01:58:49.325451 24471 catalog_manager.cc:4830] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): tablet df151ac162924af9aa78db5e4c5f9363 (table rebalancer_dynamic_table_new_name [id=525e9bc7b57942eeb740db5a17fa49da]) successfully deleted
I20240502 01:58:49.327353 31907 tablet_bootstrap.cc:492] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:49.327443 31907 ts_tablet_manager.cc:1367] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:49.328971 28447 consensus_queue.cc:1035] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.330829 31907 raft_consensus.cc:348] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.330996 31907 raft_consensus.cc:374] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.331075 31907 raft_consensus.cc:729] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.331279 31907 consensus_queue.cc:260] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.332916 31448 consensus_queue.cc:1035] T 45bfd9c929c14151b8ceeffa82d48555 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.333138 31907 ts_tablet_manager.cc:1398] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:49.333295 31907 tablet_bootstrap.cc:492] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:49.334295 31907 tablet_bootstrap.cc:654] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.346552 31932 raft_consensus.cc:482] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:49.346647 31932 raft_consensus.cc:504] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.347007 31932 leader_election.cc:288] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:49.347462 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5084f1b5d4314a5fa71e117128f0c737" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
W20240502 01:58:49.347822 27514 leader_election.cc:341] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:58:49.348081 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5084f1b5d4314a5fa71e117128f0c737" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:49.348202 27177 raft_consensus.cc:2454] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:49.348639 27513 leader_election.cc:302] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters: 1cc0a6338e0a467dbdc6efb3e67a6a09
I20240502 01:58:49.348903 31932 raft_consensus.cc:2781] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:49.348955 31932 raft_consensus.cc:482] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:49.348991 31932 raft_consensus.cc:3037] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.349506 31907 tablet_bootstrap.cc:492] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:49.349587 31907 ts_tablet_manager.cc:1367] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:49.350026 31907 raft_consensus.cc:348] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.350112 31907 raft_consensus.cc:374] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.350136 31907 raft_consensus.cc:729] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.350170 31932 raft_consensus.cc:504] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.350229 31907 consensus_queue.cc:260] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.350469 31907 ts_tablet_manager.cc:1398] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:49.350677 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5084f1b5d4314a5fa71e117128f0c737" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:49.350790 27177 raft_consensus.cc:3037] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.351562 27177 raft_consensus.cc:2454] T 5084f1b5d4314a5fa71e117128f0c737 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:49.351857 27513 leader_election.cc:302] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:49.352032 27867 raft_consensus.cc:2781] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:49.352201 27867 raft_consensus.cc:686] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:49.352376 27867 consensus_queue.cc:237] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:49.353246 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5084f1b5d4314a5fa71e117128f0c737" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:49.353343 27446 raft_consensus.cc:3037] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.354295 27446 raft_consensus.cc:2454] T 5084f1b5d4314a5fa71e117128f0c737 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:49.354899 31932 leader_election.cc:288] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:49.355571 31781 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:49.355901 31781 tablet_bootstrap.cc:492] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: Bootstrap complete.
I20240502 01:58:49.356133 31781 ts_tablet_manager.cc:1367] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.000s
I20240502 01:58:49.356216 24479 catalog_manager.cc:5484] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:49.356716 31781 raft_consensus.cc:348] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:49.356918 31781 raft_consensus.cc:729] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1049b4c8783b48f39be22bd113ad349f, State: Initialized, Role: LEARNER
I20240502 01:58:49.357084 31781 consensus_queue.cc:260] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: NON_VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { promote: true } }
I20240502 01:58:49.359344 31781 ts_tablet_manager.cc:1398] T ebe8db639839433d967583371351eacb P 1049b4c8783b48f39be22bd113ad349f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:49.360370 24784 tablet_copy_service.cc:342] P 92cb785c165749eca49af5784756461c: Request end of tablet copy session 1049b4c8783b48f39be22bd113ad349f-ebe8db639839433d967583371351eacb received from {username='slave'} at 127.17.243.6:59521
I20240502 01:58:49.360483 24784 tablet_copy_service.cc:433] P 92cb785c165749eca49af5784756461c: ending tablet copy session 1049b4c8783b48f39be22bd113ad349f-ebe8db639839433d967583371351eacb on tablet ebe8db639839433d967583371351eacb with peer 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:49.374938 27446 raft_consensus.cc:1203] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.379029 29098 raft_consensus.cc:482] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:49.379127 29098 raft_consensus.cc:504] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.379480 29098 leader_election.cc:288] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:49.379945 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad9731874143179216c33d478aa6f4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:49.380065 27446 raft_consensus.cc:2454] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:49.380311 27113 leader_election.cc:302] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:49.380419 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad9731874143179216c33d478aa6f4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:49.380486 29098 raft_consensus.cc:2781] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:49.380532 29098 raft_consensus.cc:482] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:49.380544 24896 raft_consensus.cc:2454] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:49.380565 29098 raft_consensus.cc:3037] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.381522 29098 raft_consensus.cc:504] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.381809 29098 leader_election.cc:288] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:49.382272 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad9731874143179216c33d478aa6f4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:49.382385 24896 raft_consensus.cc:3037] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.382491 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad9731874143179216c33d478aa6f4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:49.382618 27446 raft_consensus.cc:3037] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.383236 24896 raft_consensus.cc:2454] T 46ad9731874143179216c33d478aa6f4 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:49.383622 27446 raft_consensus.cc:2454] T 46ad9731874143179216c33d478aa6f4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:49.383742 27113 leader_election.cc:302] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:49.384017 29098 raft_consensus.cc:2781] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:49.384099 29098 raft_consensus.cc:686] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:49.384225 29098 consensus_queue.cc:237] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:49.386781 24479 catalog_manager.cc:5484] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:49.390646 24763 raft_consensus.cc:1203] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.397332 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:49.402709 27423 tablet_service.cc:1467] Processing CreateTablet for tablet ccafc2c89391467abaa0c45ec30f05b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]), partition=
I20240502 01:58:49.402769 24610 tablet_service.cc:1467] Processing CreateTablet for tablet ccafc2c89391467abaa0c45ec30f05b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]), partition=
I20240502 01:58:49.402948 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccafc2c89391467abaa0c45ec30f05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.403025 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccafc2c89391467abaa0c45ec30f05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.404628 31907 tablet_bootstrap.cc:492] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:49.405496 31907 tablet_bootstrap.cc:654] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.405876 31982 tablet_bootstrap.cc:492] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:49.406420 27557 tablet_service.cc:1467] Processing CreateTablet for tablet ccafc2c89391467abaa0c45ec30f05b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]), partition=
I20240502 01:58:49.406643 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccafc2c89391467abaa0c45ec30f05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:49.408254 31982 tablet_bootstrap.cc:654] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.409495 31908 tablet_bootstrap.cc:492] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:49.409848 31982 tablet_bootstrap.cc:492] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:49.409965 31982 ts_tablet_manager.cc:1367] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:49.410470 31908 tablet_bootstrap.cc:654] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:49.410408 31982 raft_consensus.cc:348] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.410513 31982 raft_consensus.cc:374] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.410543 31982 raft_consensus.cc:729] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.410665 31982 consensus_queue.cc:260] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.410938 31982 ts_tablet_manager.cc:1398] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:49.416936 31908 tablet_bootstrap.cc:492] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:49.417032 31908 ts_tablet_manager.cc:1367] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 01:58:49.417447 31908 raft_consensus.cc:348] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.417539 31908 raft_consensus.cc:374] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.417567 31908 raft_consensus.cc:729] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.417685 31908 consensus_queue.cc:260] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.417968 31908 ts_tablet_manager.cc:1398] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:49.426604 31907 tablet_bootstrap.cc:492] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:49.426689 31907 ts_tablet_manager.cc:1367] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:58:49.427129 31907 raft_consensus.cc:348] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.427222 31907 raft_consensus.cc:374] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:49.427248 31907 raft_consensus.cc:729] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:49.427367 31907 consensus_queue.cc:260] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.427630 31907 ts_tablet_manager.cc:1398] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:49.465790 27576 raft_consensus.cc:1203] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:49.476158 24763 raft_consensus.cc:1203] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:58:49.480262 31346 raft_consensus.cc:482] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:49.480356 31346 raft_consensus.cc:504] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.480690 31346 leader_election.cc:288] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:49.481489 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dba43199ac941dcaefe4f0236e90156" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:49.481551 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dba43199ac941dcaefe4f0236e90156" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f" is_pre_election: true
I20240502 01:58:49.481621 25034 raft_consensus.cc:2454] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:49.481693 27312 raft_consensus.cc:2454] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:49.482385 27377 leader_election.cc:302] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1049b4c8783b48f39be22bd113ad349f, 1cc0a6338e0a467dbdc6efb3e67a6a09; no voters:
I20240502 01:58:49.483831 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49994
I20240502 01:58:49.483964 24896 raft_consensus.cc:595] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:49.485553 31346 raft_consensus.cc:2781] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:49.485615 31346 raft_consensus.cc:482] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:49.485659 31346 raft_consensus.cc:3037] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.486773 31346 raft_consensus.cc:504] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.487094 31346 leader_election.cc:288] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:49.487615 25034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dba43199ac941dcaefe4f0236e90156" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:49.487717 25034 raft_consensus.cc:3037] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.487970 27312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dba43199ac941dcaefe4f0236e90156" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049b4c8783b48f39be22bd113ad349f"
I20240502 01:58:49.488083 27312 raft_consensus.cc:3037] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.488628 25034 raft_consensus.cc:2454] T 0dba43199ac941dcaefe4f0236e90156 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:49.488968 27377 leader_election.cc:302] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:49.489065 27312 raft_consensus.cc:2454] T 0dba43199ac941dcaefe4f0236e90156 P 1049b4c8783b48f39be22bd113ad349f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:49.489145 31346 raft_consensus.cc:2781] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:49.489209 31346 raft_consensus.cc:686] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:49.489320 31346 consensus_queue.cc:237] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:49.491428 24479 catalog_manager.cc:5484] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:49.522061 28447 raft_consensus.cc:482] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:49.522166 28447 raft_consensus.cc:504] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.522527 28447 leader_election.cc:288] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:49.522987 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccafc2c89391467abaa0c45ec30f05b2" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:49.523029 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccafc2c89391467abaa0c45ec30f05b2" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:49.523116 24632 raft_consensus.cc:2454] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:49.523176 27576 raft_consensus.cc:2454] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:49.523447 27380 leader_election.cc:302] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:49.523662 28447 raft_consensus.cc:2781] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:49.523716 28447 raft_consensus.cc:482] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:49.523753 28447 raft_consensus.cc:3037] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.524535 28447 raft_consensus.cc:504] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.524863 28447 leader_election.cc:288] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:49.525146 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccafc2c89391467abaa0c45ec30f05b2" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:49.525240 27576 raft_consensus.cc:3037] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.525849 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccafc2c89391467abaa0c45ec30f05b2" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:49.526070 27576 raft_consensus.cc:2454] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:49.526089 24632 raft_consensus.cc:3037] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:49.526340 27380 leader_election.cc:302] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:49.526540 28447 raft_consensus.cc:2781] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:49.526625 28447 raft_consensus.cc:686] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:49.526774 28447 consensus_queue.cc:237] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:49.527001 24632 raft_consensus.cc:2454] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:49.528867 24479 catalog_manager.cc:5484] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:49.558167 25034 raft_consensus.cc:1203] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.575862 24896 raft_consensus.cc:1203] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.578864 31814 raft_consensus.cc:979] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: : Instructing follower 9dd619a595e94004ac0f7f5a9054e7cc to start an election
I20240502 01:58:49.578955 31814 raft_consensus.cc:1067] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Signalling peer 9dd619a595e94004ac0f7f5a9054e7cc to start an election
I20240502 01:58:49.579509 27177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d"
dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
from {username='slave'} at 127.17.243.3:55443
I20240502 01:58:49.579665 27177 raft_consensus.cc:482] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:49.579716 27177 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:49.580776 27177 raft_consensus.cc:504] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:49.581180 27177 leader_election.cc:288] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:49.581918 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:49.582080 24763 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:49.582980 24763 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 2.
I20240502 01:58:49.583492 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:49.583608 27445 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:49.584298 27445 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 2.
I20240502 01:58:49.584621 27113 leader_election.cc:302] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:49.584724 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0d9525b5f4384861dc50d0bf4967d" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:49.584793 31227 raft_consensus.cc:2781] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:49.584822 24896 raft_consensus.cc:3032] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:49.584857 24896 raft_consensus.cc:729] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:49.584867 31227 raft_consensus.cc:686] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:49.584975 24896 consensus_queue.cc:260] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:49.585021 31227 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:49.585130 24896 raft_consensus.cc:3037] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:49.585953 24896 raft_consensus.cc:2454] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 2.
I20240502 01:58:49.587028 24486 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 1 to 2, leader changed from f46bcd2bc124471981349b1e2726253b (127.17.243.3) to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 2 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:49.599475 24896 raft_consensus.cc:1203] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.602553 27445 raft_consensus.cc:1203] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:58:49.625802 27312 raft_consensus.cc:1203] T 89580549c13d4c1ebdb2a5d24da2a01e P 1049b4c8783b48f39be22bd113ad349f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:58:49.625977 24763 raft_consensus.cc:1203] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:49.643252 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:49.646656 24632 raft_consensus.cc:1261] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:49.646694 27576 raft_consensus.cc:1261] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:49.647030 31346 consensus_queue.cc:1035] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.647142 28447 consensus_queue.cc:1035] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.650200 24486 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]). Expected version 1 got 0
I20240502 01:58:49.650622 31781 ts_tablet_manager.cc:902] T 9b8667a49f1a44339122040b0122be0d P 1049b4c8783b48f39be22bd113ad349f: Initiating tablet copy from peer e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:49.650619 24481 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]). Expected version 1 got 0
F20240502 01:58:49.650799 31781 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e47667fc8fa49e6b18ee3df06cda00a.queue_time_us, tablet-download-pool-0e47667fc8fa49e6b18ee3df06cda00a.queue_time_us) (tablet-download-pool-0e47667fc8fa49e6b18ee3df06cda00a.run_cpu_time_us, tablet-download-pool-0e47667fc8fa49e6b18ee3df06cda00a.run_cpu_time_us) (tablet-download-pool-0e47667fc8fa49e6b18ee3df06cda00a.run_wall_time_us, tablet-download-pool-0e47667fc8fa49e6b18ee3df06cda00a.run_wall_time_us) (tablet-download-pool-16ae91433f10465c922183298a387caa.queue_time_us, tablet-download-pool-16ae91433f10465c922183298a387caa.queue_time_us) (tablet-download-pool-16ae91433f10465c922183298a387caa.run_cpu_time_us, tablet-download-pool-16ae91433f10465c922183298a387caa.run_cpu_time_us) (tablet-download-pool-16ae91433f10465c922183298a387caa.run_wall_time_us, tablet-download-pool-16ae91433f10465c922183298a387caa.run_wall_time_us) (tablet-download-pool-26b7501639064035a570b3b4c1cf083f.queue_time_us, tablet-download-pool-26b7501639064035a570b3b4c1cf083f.queue_time_us) (tablet-download-pool-26b7501639064035a570b3b4c1cf083f.run_cpu_time_us, tablet-download-pool-26b7501639064035a570b3b4c1cf083f.run_cpu_time_us) (tablet-download-pool-26b7501639064035a570b3b4c1cf083f.run_wall_time_us, tablet-download-pool-26b7501639064035a570b3b4c1cf083f.run_wall_time_us) (tablet-download-pool-48085cebf94e4f549e6174a88acdcc59.queue_time_us, tablet-download-pool-48085cebf94e4f549e6174a88acdcc59.queue_time_us) (tablet-download-pool-48085cebf94e4f549e6174a88acdcc59.run_cpu_time_us, tablet-download-pool-48085cebf94e4f549e6174a88acdcc59.run_cpu_time_us) (tablet-download-pool-48085cebf94e4f549e6174a88acdcc59.run_wall_time_us, tablet-download-pool-48085cebf94e4f549e6174a88acdcc59.run_wall_time_us) (tablet-download-pool-693e0959e61c44c59f797e4060187cef.queue_time_us, tablet-download-pool-693e0959e61c44c59f797e4060187cef.queue_time_us) (tablet-download-pool-693e0959e61c44c59f797e4060187cef.run_cpu_time_us, tablet-download-pool-693e0959e61c44c59f797e4060187cef.run_cpu_time_us) (tablet-download-pool-693e0959e61c44c59f797e4060187cef.run_wall_time_us, tablet-download-pool-693e0959e61c44c59f797e4060187cef.run_wall_time_us) (tablet-download-pool-7d0cc9aba3804968a6dce97682f6b38e.queue_time_us, tablet-download-pool-7d0cc9aba3804968a6dce97682f6b38e.queue_time_us) (tablet-download-pool-7d0cc9aba3804968a6dce97682f6b38e.run_cpu_time_us, tablet-download-pool-7d0cc9aba3804968a6dce97682f6b38e.run_cpu_time_us) (tablet-download-pool-7d0cc9aba3804968a6dce97682f6b38e.run_wall_time_us, tablet-download-pool-7d0cc9aba3804968a6dce97682f6b38e.run_wall_time_us) (tablet-download-pool-89580549c13d4c1ebdb2a5d24da2a01e.queue_time_us, tablet-download-pool-89580549c13d4c1ebdb2a5d24da2a01e.queue_time_us) (tablet-download-pool-89580549c13d4c1ebdb2a5d24da2a01e.run_cpu_time_us, tablet-download-pool-89580549c13d4c1ebdb2a5d24da2a01e.run_cpu_time_us) (tablet-download-pool-89580549c13d4c1ebdb2a5d24da2a01e.run_wall_time_us, tablet-download-pool-89580549c13d4c1ebdb2a5d24da2a01e.run_wall_time_us) (tablet-download-pool-92382869887b43f9a23a2c0ea0012720.queue_time_us, tablet-download-pool-92382869887b43f9a23a2c0ea0012720.queue_time_us) (tablet-download-pool-92382869887b43f9a23a2c0ea0012720.run_cpu_time_us, tablet-download-pool-92382869887b43f9a23a2c0ea0012720.run_cpu_time_us) (tablet-download-pool-92382869887b43f9a23a2c0ea0012720.run_wall_time_us, tablet-download-pool-92382869887b43f9a23a2c0ea0012720.run_wall_time_us) (tablet-download-pool-938581084b27405cbc01917e1a982208.queue_time_us, tablet-download-pool-938581084b27405cbc01917e1a982208.queue_time_us) (tablet-download-pool-938581084b27405cbc01917e1a982208.run_cpu_time_us, tablet-download-pool-938581084b27405cbc01917e1a982208.run_cpu_time_us) (tablet-download-pool-938581084b27405cbc01917e1a982208.run_wall_time_us, tablet-download-pool-938581084b27405cbc01917e1a982208.run_wall_time_us) (tablet-download-pool-9b8667a49f1a44339122040b0122be0d.queue_time_us, tablet-download-pool-9b8667a49f1a44339122040b0122be0d.queue_time_us) (tablet-download-pool-9ede6fc65d044804848e6f61e8c238c0.queue_time_us, tablet-download-pool-9ede6fc65d044804848e6f61e8c238c0.queue_time_us) (tablet-download-pool-9ede6fc65d044804848e6f61e8c238c0.run_cpu_time_us, tablet-download-pool-9ede6fc65d044804848e6f61e8c238c0.run_cpu_time_us) (tablet-download-pool-9ede6fc65d044804848e6f61e8c238c0.run_wall_time_us, tablet-download-pool-9ede6fc65d044804848e6f61e8c238c0.run_wall_time_us) (tablet-download-pool-a094659d539e4c4f9861e356168c5894.queue_time_us, tablet-download-pool-a094659d539e4c4f9861e356168c5894.queue_time_us) (tablet-download-pool-a094659d539e4c4f9861e356168c5894.run_cpu_time_us, tablet-download-pool-a094659d539e4c4f9861e356168c5894.run_cpu_time_us) (tablet-download-pool-a094659d539e4c4f9861e356168c5894.run_wall_time_us, tablet-download-pool-a094659d539e4c4f9861e356168c5894.run_wall_time_us) (tablet-download-pool-a6846b4da6654802b00203529a01e398.queue_time_us, tablet-download-pool-a6846b4da6654802b00203529a01e398.queue_time_us) (tablet-download-pool-a6846b4da6654802b00203529a01e398.run_cpu_time_us, tablet-download-pool-a6846b4da6654802b00203529a01e398.run_cpu_time_us) (tablet-download-pool-a6846b4da6654802b00203529a01e398.run_wall_time_us, tablet-download-pool-a6846b4da6654802b00203529a01e398.run_wall_time_us) (tablet-download-pool-b3edec06b3d8450a8cc3002be70c848a.queue_time_us, tablet-download-pool-b3edec06b3d8450a8cc3002be70c848a.queue_time_us) (tablet-download-pool-b3edec06b3d8450a8cc3002be70c848a.run_cpu_time_us, tablet-download-pool-b3edec06b3d8450a8cc3002be70c848a.run_cpu_time_us) (tablet-download-pool-b3edec06b3d8450a8cc3002be70c848a.run_wall_time_us, tablet-download-pool-b3edec06b3d8450a8cc3002be70c848a.run_wall_time_us) (tablet-download-pool-ebe8db639839433d967583371351eacb.queue_time_us, tablet-download-pool-ebe8db639839433d967583371351eacb.queue_time_us) (tablet-download-pool-ebe8db639839433d967583371351eacb.run_cpu_time_us, tablet-download-pool-ebe8db639839433d967583371351eacb.run_cpu_time_us) (tablet-download-pool-ebe8db639839433d967583371351eacb.run_wall_time_us, tablet-download-pool-ebe8db639839433d967583371351eacb.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 01:58:49.651413 32035 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:49.651728 32035 tablet.cc:1620] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:49.653821 24896 raft_consensus.cc:1203] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7fd8500bfdcd google::LogMessage::Fail() at ??:0
I20240502 01:58:49.656419 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978] alter complete (version 1)
I20240502 01:58:49.657737 24481 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]). Expected version 1 got 0
@ 0x7fd8500c3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd8500bf7cc google::LogMessage::Flush() at ??:0
I20240502 01:58:49.660342 32037 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:49.660624 32037 tablet.cc:1620] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:49.661192 24479 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978]). Expected version 1 got 0
@ 0x7fd8500c0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:58:49.662261 32034 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:49.662552 32034 tablet.cc:1620] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
@ 0x7fd851529044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd85150df98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fd85150befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fd8540be77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fd8540c7a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fd85414876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fd85415c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fd854147226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fd854156c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559e9238f75c std::function<>::operator()() at ??:0
@ 0x7fd851510e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd8515116ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd85151303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559e9238f75c std::function<>::operator()() at ??:0
@ 0x7fd851500807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd851d3e6db start_thread at ??:0
@ 0x7fd84f27671f clone at ??:0
I20240502 01:58:49.743057 30520 raft_consensus.cc:1050] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc: attempting to promote NON_VOTER 588bfc090a5247598cd5d3cee5ad2007 to VOTER
I20240502 01:58:49.743460 30520 consensus_queue.cc:237] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } }
I20240502 01:58:49.745098 24632 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEARNER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.745296 27576 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.745756 29098 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.745963 27445 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.746006 32084 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.748049 29098 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.749763 32084 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.750288 27576 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.752451 24632 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.754834 24479 catalog_manager.cc:5484] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.756131 27445 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.760046 27177 consensus_queue.cc:237] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } }
I20240502 01:58:49.761468 27445 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.762583 30520 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.763106 24632 raft_consensus.cc:1261] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.763453 30520 consensus_queue.cc:1035] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.764048 32084 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.764430 27445 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.766419 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.767452 24481 catalog_manager.cc:5484] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.768790 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:49.770298 24632 raft_consensus.cc:2932] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { promote: false } } }
I20240502 01:58:49.772246 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 with delete_type TABLET_DATA_TOMBSTONED (TS bb653087e8554c7591cf547dd969f3be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:49.772608 32091 tablet_replica.cc:330] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:49.772745 32091 raft_consensus.cc:2229] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.772869 32091 raft_consensus.cc:2258] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.773430 32091 ts_tablet_manager.cc:1875] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.776875 32091 ts_tablet_manager.cc:1888] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:49.776943 32091 log.cc:1196] T 3c8e4bca52cc47e89ad2b02d5035b1b7 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/3c8e4bca52cc47e89ad2b02d5035b1b7
I20240502 01:58:49.777448 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 3c8e4bca52cc47e89ad2b02d5035b1b7 (table rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]) successfully deleted
I20240502 01:58:49.778316 32034 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:49.778610 32034 tablet.cc:1620] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:49.778868 32035 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:49.779112 32035 tablet.cc:1620] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:49.783926 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=2724e9ee91654ae5af45296582daf978] alter complete (version 2)
I20240502 01:58:49.784909 32037 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:49.785197 32037 tablet.cc:1620] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:49.819401 32084 consensus_queue.cc:1035] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.823266 29098 consensus_queue.cc:1035] T 46ad9731874143179216c33d478aa6f4 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.831944 31949 consensus_queue.cc:1035] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.834465 27867 consensus_queue.cc:1035] T 5084f1b5d4314a5fa71e117128f0c737 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.861722 31949 raft_consensus.cc:1050] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:49.862188 31949 consensus_queue.cc:237] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:49.863526 27445 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEARNER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.863780 24632 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.863780 27177 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.864154 31932 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.864303 27867 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.864778 31949 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.866936 27867 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.867341 27445 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.868770 24632 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.870455 31448 raft_consensus.cc:1050] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09: attempting to promote NON_VOTER 92cb785c165749eca49af5784756461c to VOTER
I20240502 01:58:49.870821 31448 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:49.871631 31932 raft_consensus.cc:1050] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be: attempting to promote NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 to VOTER
I20240502 01:58:49.871726 24481 catalog_manager.cc:5484] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.871968 31932 consensus_queue.cc:237] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:49.873397 24632 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.873801 25032 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.873703 27177 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.874222 31949 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.875404 31346 consensus_queue.cc:1035] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:49.875792 25034 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:49.877270 32100 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.882141 31346 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:49.883318 27576 consensus_queue.cc:237] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:49.883574 24763 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:49.883633 27445 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.883641 27578 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:49.884222 31346 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:49.884397 31949 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.884541 31346 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:49.885403 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:49.885921 32107 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.887223 24632 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.887684 31932 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.888190 25034 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.888367 24632 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.891183 24481 catalog_manager.cc:5484] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.894668 27445 raft_consensus.cc:1261] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.895483 27867 consensus_queue.cc:1035] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.899340 32034 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:49.900346 32037 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 01:58:49.902880 24564 connection.cc:535] client connection to 127.17.243.6:41707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:58:49.903020 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:58:49.903353 27377 connection.cc:535] client connection to 127.17.243.6:41707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:58:49.903465 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:58:49.903578 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.243.6:41707 (error 108)
W20240502 01:58:49.904000 27110 connection.cc:535] client connection to 127.17.243.6:41707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:58:49.904112 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:58:49.905748 32120 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:58:49.904704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.243.6:41707 (local address 127.17.243.5:37701)
0502 01:58:49.905034 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0502 01:58:49.905049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 01:58:49.905130 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:58:49.905230 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
W20240502 01:58:49.907116 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.907752 32107 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.908279 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=2724e9ee91654ae5af45296582daf978] alter complete (version 3)
I20240502 01:58:49.909173 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet abfa462d7c6d43f0ac967d298ee9050e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.909657 24632 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.909858 27445 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.911679 32035 tablet.cc:1722] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:49.912010 27445 raft_consensus.cc:2932] T abfa462d7c6d43f0ac967d298ee9050e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.913977 24482 catalog_manager.cc:5484] T abfa462d7c6d43f0ac967d298ee9050e P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 3 to 4, VOTER 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5) evicted. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:49.915287 27578 consensus_queue.cc:237] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:49.916802 27155 tablet_service.cc:1514] Processing DeleteTablet for tablet abfa462d7c6d43f0ac967d298ee9050e with delete_type TABLET_DATA_TOMBSTONED (TS 9dd619a595e94004ac0f7f5a9054e7cc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:49.916985 25031 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.917379 27445 raft_consensus.cc:1261] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.917397 31932 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.917733 32100 consensus_queue.cc:1035] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:58:49.919848 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:49.920497 31932 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.920509 32105 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
W20240502 01:58:49.921046 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:49.921110 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.921875 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2e41964ae354b4d92385663ae940bc0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.921911 25032 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
W20240502 01:58:49.922343 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:49.922322 25031 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.924714 24763 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.925285 32125 tablet_replica.cc:330] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:49.925431 32125 raft_consensus.cc:2229] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.925630 32125 raft_consensus.cc:2258] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.926877 27576 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.927186 24482 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 5 to 6, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.927795 27445 raft_consensus.cc:2932] T b2e41964ae354b4d92385663ae940bc0 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:49.926431 32125 ts_tablet_manager.cc:1875] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.930831 24481 catalog_manager.cc:5484] T b2e41964ae354b4d92385663ae940bc0 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 3 to 4, VOTER 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1) evicted. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:49.933079 32125 ts_tablet_manager.cc:1888] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:49.933259 32125 log.cc:1196] T abfa462d7c6d43f0ac967d298ee9050e P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/abfa462d7c6d43f0ac967d298ee9050e
I20240502 01:58:49.934062 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e41964ae354b4d92385663ae940bc0 with delete_type TABLET_DATA_TOMBSTONED (TS 588bfc090a5247598cd5d3cee5ad2007 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:49.934069 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet abfa462d7c6d43f0ac967d298ee9050e (table rebalancer_extra_table_8 [id=63c3dce291044cbb8ba3bbf9b783a011]) successfully deleted
I20240502 01:58:49.934386 32131 tablet_replica.cc:330] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:49.934515 32131 raft_consensus.cc:2229] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.934643 32131 raft_consensus.cc:2258] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:58:49.934690 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:49.935583 32131 ts_tablet_manager.cc:1875] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.937551 27445 consensus_queue.cc:237] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:49.937858 32131 ts_tablet_manager.cc:1888] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:49.938076 32131 log.cc:1196] T b2e41964ae354b4d92385663ae940bc0 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/b2e41964ae354b4d92385663ae940bc0
I20240502 01:58:49.938660 25031 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:49.938685 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet b2e41964ae354b4d92385663ae940bc0 (table rebalancer_extra_table_16 [id=b57c683f6f464fa9b9dc3161d01ec277]) successfully deleted
I20240502 01:58:49.939227 24763 raft_consensus.cc:1261] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:49.939740 31448 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:49.940027 31448 consensus_queue.cc:1035] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:49.941584 31448 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.942984 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adc992deb712458aa793eaa11019de1f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.943228 24763 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.943874 25031 raft_consensus.cc:2932] T adc992deb712458aa793eaa11019de1f P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.946465 24481 catalog_manager.cc:5484] T adc992deb712458aa793eaa11019de1f P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 6 to 7, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
W20240502 01:58:49.947840 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.949196 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet adc992deb712458aa793eaa11019de1f with delete_type TABLET_DATA_TOMBSTONED (TS bb653087e8554c7591cf547dd969f3be not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:49.949407 32091 tablet_replica.cc:330] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:49.949540 32091 raft_consensus.cc:2229] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.949715 32091 raft_consensus.cc:2258] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.950263 32091 ts_tablet_manager.cc:1875] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.951836 32091 ts_tablet_manager.cc:1888] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:58:49.951896 32091 log.cc:1196] T adc992deb712458aa793eaa11019de1f P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/adc992deb712458aa793eaa11019de1f
I20240502 01:58:49.952481 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet adc992deb712458aa793eaa11019de1f (table rebalancer_extra_table_6 [id=10b365e99dae47b3a0e2952c284733e4]) successfully deleted
W20240502 01:58:49.958211 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:49.958298 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.964222 31348 raft_consensus.cc:1050] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:49.964628 31348 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:49.965441 32105 raft_consensus.cc:1050] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09: attempting to promote NON_VOTER 92cb785c165749eca49af5784756461c to VOTER
I20240502 01:58:49.965837 32105 consensus_queue.cc:237] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:49.966126 25031 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.966215 27576 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.966595 31411 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.967571 27177 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:49.967651 24763 raft_consensus.cc:1261] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 01:58:49.967803 27377 consensus_peers.cc:485] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.968056 31411 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.968607 32105 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:58:49.969235 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.969915 27577 raft_consensus.cc:1261] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.970134 31411 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:49.971511 32105 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.972458 31411 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.973385 27177 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.974150 27576 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.975711 25031 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.975939 32105 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.976296 24479 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.978775 24482 catalog_manager.cc:5484] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:49.980826 24765 consensus_queue.cc:237] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:49.980886 24763 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.982086 27445 consensus_queue.cc:237] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:49.982721 27577 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:49.983232 31932 raft_consensus.cc:1050] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:49.983282 31348 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:49.983388 27576 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.983583 31932 consensus_queue.cc:237] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:49.983903 27177 raft_consensus.cc:1261] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:58:49.984283 31348 consensus_queue.cc:1035] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:49.984973 27577 raft_consensus.cc:1261] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.985292 24897 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.985504 24765 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.985852 32147 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.986263 31448 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.986404 27445 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:49.987500 32100 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.987672 27577 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.987815 32100 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.988101 32100 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:49.988333 27177 raft_consensus.cc:2932] T 63312ec15b034e47bc2dd484b57d4648 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:49.989126 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63312ec15b034e47bc2dd484b57d4648 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.990119 24482 catalog_manager.cc:5484] T 63312ec15b034e47bc2dd484b57d4648 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 8 to 9, VOTER e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
W20240502 01:58:49.991194 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:49.991997 31448 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.993384 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 63312ec15b034e47bc2dd484b57d4648 with delete_type TABLET_DATA_TOMBSTONED (TS e0b9e7df4e8543fa9b8e3ffc24330657 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:49.994081 24763 raft_consensus.cc:1261] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:49.994138 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b2686f7b58341e9a0b1f2b27cfbb173 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:49.994469 32105 consensus_queue.cc:1035] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:49.994585 27577 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.994810 32163 tablet_replica.cc:330] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:49.994935 32163 raft_consensus.cc:2229] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.995141 32163 raft_consensus.cc:2258] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.995358 24763 raft_consensus.cc:2932] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:49.996053 32163 ts_tablet_manager.cc:1875] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.997325 24482 catalog_manager.cc:5484] T 9b2686f7b58341e9a0b1f2b27cfbb173 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:49.998557 31411 raft_consensus.cc:1050] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER e0b9e7df4e8543fa9b8e3ffc24330657 to VOTER
I20240502 01:58:49.998615 32163 ts_tablet_manager.cc:1888] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:58:49.998682 32163 log.cc:1196] T 63312ec15b034e47bc2dd484b57d4648 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/63312ec15b034e47bc2dd484b57d4648
I20240502 01:58:49.998893 31411 consensus_queue.cc:237] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:49.999209 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 63312ec15b034e47bc2dd484b57d4648 (table rebalance_test_table_13 [id=f5b9047ae09b449c8fbaccaeacda8c32]) successfully deleted
I20240502 01:58:50.000175 25031 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.000471 27177 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.000617 31348 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:50.001081 31348 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:50.001514 27577 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.003687 27867 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
W20240502 01:58:50.004117 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.004089 24897 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.005411 27445 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.005769 32147 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:58:50.006314 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.006356 24486 catalog_manager.cc:5484] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:50.008427 24765 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.008900 27445 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:50.009217 29098 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:50.009281 24897 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:50.009677 31348 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.011356 27177 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.011518 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:50.011612 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:50.012925 32084 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:50.012948 27578 consensus_queue.cc:237] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:50.013767 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet ccafc2c89391467abaa0c45ec30f05b2
I20240502 01:58:50.014568 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet ccafc2c89391467abaa0c45ec30f05b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:50 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:50.015241 24481 catalog_manager.cc:5484] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.015810 25031 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.015938 24897 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:50.016984 31949 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:50.018194 24765 raft_consensus.cc:1261] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:50.018388 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet ccafc2c89391467abaa0c45ec30f05b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:50 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:50.018419 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet ccafc2c89391467abaa0c45ec30f05b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:50 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:50.018558 32091 tablet_replica.cc:330] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:50.018630 32131 tablet_replica.cc:330] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:50.018678 32091 raft_consensus.cc:2229] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.018743 32131 raft_consensus.cc:2229] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.018769 32091 raft_consensus.cc:2258] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.018849 32131 raft_consensus.cc:2258] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.019383 32091 ts_tablet_manager.cc:1875] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:50.019826 32131 ts_tablet_manager.cc:1875] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 01:58:50.021195 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b2686f7b58341e9a0b1f2b27cfbb173: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.021303 32179 tablet_replica.cc:330] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:50.021335 32091 ts_tablet_manager.cc:1888] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:50.021391 32091 log.cc:1196] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/ccafc2c89391467abaa0c45ec30f05b2
I20240502 01:58:50.021430 32179 raft_consensus.cc:2229] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:50.021562 32179 raft_consensus.cc:2258] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.021621 32091 ts_tablet_manager.cc:1909] T ccafc2c89391467abaa0c45ec30f05b2 P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:50.022224 32179 ts_tablet_manager.cc:1875] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:50.022439 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table_new_name [id=2724e9ee91654ae5af45296582daf978]) successfully deleted
I20240502 01:58:50.022620 32107 consensus_queue.cc:1035] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:50.023036 32100 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.024631 24896 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.024737 32179 ts_tablet_manager.cc:1888] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:50.024801 32179 log.cc:1196] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/ccafc2c89391467abaa0c45ec30f05b2
I20240502 01:58:50.025029 32179 ts_tablet_manager.cc:1909] T ccafc2c89391467abaa0c45ec30f05b2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:50.025475 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60a07252321c4a0ebc16023c49015ffd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:50.027019 24483 catalog_manager.cc:5484] T 60a07252321c4a0ebc16023c49015ffd P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:50.027376 24765 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:50.027740 30520 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:50.028074 27577 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.029282 24764 consensus_queue.cc:237] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } }
I20240502 01:58:50.030766 27177 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:50.030808 25031 raft_consensus.cc:1261] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:50.031311 32147 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:50.031504 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a07252321c4a0ebc16023c49015ffd with delete_type TABLET_DATA_TOMBSTONED (TS 1cc0a6338e0a467dbdc6efb3e67a6a09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:50.032287 31457 consensus_queue.cc:1035] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:50.032301 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:50.033581 32147 raft_consensus.cc:1050] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: attempting to promote NON_VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 to VOTER
I20240502 01:58:50.033985 32147 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:50.034318 31411 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.034893 27177 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.035543 27577 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:50.035578 27445 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 LEARNER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:50.035758 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b066a49a2b444d3961cac30b5cb6adf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:50.035897 31457 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:50.036643 24632 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:50.036576 25031 raft_consensus.cc:2932] T 9b066a49a2b444d3961cac30b5cb6adf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } } }
I20240502 01:58:50.036724 32131 ts_tablet_manager.cc:1888] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:50.036793 32131 log.cc:1196] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/ccafc2c89391467abaa0c45ec30f05b2
I20240502 01:58:50.037026 32131 ts_tablet_manager.cc:1909] T ccafc2c89391467abaa0c45ec30f05b2 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:50.037036 31457 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:50.037658 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table_new_name [id=2724e9ee91654ae5af45296582daf978]) successfully deleted
W20240502 01:58:50.038074 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.038776 32147 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:58:50.039875 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.041252 32179 tablet_replica.cc:330] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:50.041389 31227 raft_consensus.cc:1050] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc: attempting to promote NON_VOTER 92cb785c165749eca49af5784756461c to VOTER
I20240502 01:58:50.041396 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet ccafc2c89391467abaa0c45ec30f05b2 (table rebalancer_dynamic_table_new_name [id=2724e9ee91654ae5af45296582daf978]) successfully deleted
I20240502 01:58:50.041776 31227 consensus_queue.cc:237] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:50.042423 27155 tablet_service.cc:1467] Processing CreateTablet for tablet 252502ad2e124996b345888706e674d5 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.042663 27155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252502ad2e124996b345888706e674d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.042968 24481 catalog_manager.cc:5484] T 9b066a49a2b444d3961cac30b5cb6adf P 92cb785c165749eca49af5784756461c reported cstate change: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:50.043092 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 7db1384cb0374899a6e17a550e1f7cba (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.043318 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db1384cb0374899a6e17a550e1f7cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.043879 24765 raft_consensus.cc:1261] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.044221 30520 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:50.044826 27156 tablet_service.cc:1467] Processing CreateTablet for tablet e28b2f0e50da4719aabefb28c34c2a0a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.045051 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28b2f0e50da4719aabefb28c34c2a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.045786 27578 raft_consensus.cc:1261] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.046181 30520 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:58:50.046917 27110 consensus_peers.cc:485] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.047399 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 252502ad2e124996b345888706e674d5 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.047621 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252502ad2e124996b345888706e674d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.048142 31457 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
W20240502 01:58:50.048902 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.050213 27577 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:50.050761 27557 tablet_service.cc:1467] Processing CreateTablet for tablet e28b2f0e50da4719aabefb28c34c2a0a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.051010 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28b2f0e50da4719aabefb28c34c2a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.050920 27445 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:50.050761 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 7db1384cb0374899a6e17a550e1f7cba (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.051359 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db1384cb0374899a6e17a550e1f7cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.052471 24877 tablet_service.cc:1467] Processing CreateTablet for tablet e28b2f0e50da4719aabefb28c34c2a0a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=46ea81fc3663418f8752178c50a81751]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.052718 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28b2f0e50da4719aabefb28c34c2a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.053337 32179 raft_consensus.cc:2229] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.054147 32188 tablet_bootstrap.cc:492] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:50.054109 24632 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:50.054597 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b066a49a2b444d3961cac30b5cb6adf with delete_type TABLET_DATA_TOMBSTONED (TS bb653087e8554c7591cf547dd969f3be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:50.054528 24763 raft_consensus.cc:2932] T 60a07252321c4a0ebc16023c49015ffd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.054780 32091 tablet_replica.cc:330] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:50.054900 32091 raft_consensus.cc:2229] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.055032 32091 raft_consensus.cc:2258] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.056030 32188 tablet_bootstrap.cc:654] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.057318 32091 ts_tablet_manager.cc:1875] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:50.057968 24486 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 8 to 9, 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:50.059378 32091 ts_tablet_manager.cc:1888] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:50.059473 32091 log.cc:1196] T 9b066a49a2b444d3961cac30b5cb6adf P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/9b066a49a2b444d3961cac30b5cb6adf
I20240502 01:58:50.060005 32195 tablet_bootstrap.cc:492] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:50.060225 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 9b066a49a2b444d3961cac30b5cb6adf (table rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]) successfully deleted
I20240502 01:58:50.062191 32192 tablet_bootstrap.cc:492] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:50.053481 32179 raft_consensus.cc:2258] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.055269 32191 tablet_bootstrap.cc:492] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:50.063166 32192 tablet_bootstrap.cc:654] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.063421 32179 ts_tablet_manager.cc:1875] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:50.063558 32191 tablet_bootstrap.cc:654] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.064347 32188 tablet_bootstrap.cc:492] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:50.064471 32188 ts_tablet_manager.cc:1367] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:50.064977 32188 raft_consensus.cc:348] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.065100 32188 raft_consensus.cc:374] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.065156 32188 raft_consensus.cc:729] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.065435 32179 ts_tablet_manager.cc:1888] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20240502 01:58:50.065631 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.065577 29098 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.065943 32188 consensus_queue.cc:260] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.066228 32188 ts_tablet_manager.cc:1398] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:50.066121 27578 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.066358 32188 tablet_bootstrap.cc:492] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:50.067178 32188 tablet_bootstrap.cc:654] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.067524 24765 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.068569 32195 tablet_bootstrap.cc:654] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.068711 24483 catalog_manager.cc:5484] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.070155 32192 tablet_bootstrap.cc:492] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:50.070245 32192 ts_tablet_manager.cc:1367] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:50.070662 32192 raft_consensus.cc:348] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.067320 32191 tablet_bootstrap.cc:492] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:50.071403 32191 ts_tablet_manager.cc:1367] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:50.070775 32192 raft_consensus.cc:374] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.071558 32192 raft_consensus.cc:729] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.071696 32192 consensus_queue.cc:260] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.071979 32192 ts_tablet_manager.cc:1398] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:50.072198 32192 tablet_bootstrap.cc:492] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:50.072144 32191 raft_consensus.cc:348] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.072239 32191 raft_consensus.cc:374] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.072269 32191 raft_consensus.cc:729] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.072258 27177 consensus_queue.cc:237] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:50.072376 32191 consensus_queue.cc:260] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.072659 32191 ts_tablet_manager.cc:1398] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.074069 24765 raft_consensus.cc:1261] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:50.074448 27578 raft_consensus.cc:1261] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:50.074455 32192 tablet_bootstrap.cc:654] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.074496 31227 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:50.074919 31227 consensus_queue.cc:1035] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:50.075028 32179 log.cc:1196] T 60a07252321c4a0ebc16023c49015ffd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/60a07252321c4a0ebc16023c49015ffd
I20240502 01:58:50.075989 32195 tablet_bootstrap.cc:492] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:50.076128 32195 ts_tablet_manager.cc:1367] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 01:58:50.076128 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 60a07252321c4a0ebc16023c49015ffd (table rebalancer_extra_table_14 [id=e9d5cfbf48a64c2f9be8288c45ccc447]) successfully deleted
I20240502 01:58:50.076634 32195 raft_consensus.cc:348] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.076795 32195 raft_consensus.cc:374] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.076843 32195 raft_consensus.cc:729] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.076972 32195 consensus_queue.cc:260] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.077739 30520 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.077880 32195 ts_tablet_manager.cc:1398] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:50.078799 32188 tablet_bootstrap.cc:492] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:50.078884 32188 ts_tablet_manager.cc:1367] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:58:50.079119 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c97e5f78b03245cc86f70b1d50fcf723 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:50.079463 32188 raft_consensus.cc:348] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.079551 32188 raft_consensus.cc:374] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.079578 32188 raft_consensus.cc:729] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.079555 27577 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.079689 32188 consensus_queue.cc:260] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.079950 32188 ts_tablet_manager.cc:1398] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:50.081413 24765 raft_consensus.cc:2932] T c97e5f78b03245cc86f70b1d50fcf723 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:50.082479 32188 tablet_bootstrap.cc:492] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:50.082451 24481 catalog_manager.cc:5484] T c97e5f78b03245cc86f70b1d50fcf723 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 3 to 4, VOTER 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) evicted. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:50.083402 32188 tablet_bootstrap.cc:654] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.085600 32188 tablet_bootstrap.cc:492] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:50.085997 32188 ts_tablet_manager.cc:1367] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:50.086419 32188 raft_consensus.cc:348] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.086504 32188 raft_consensus.cc:374] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.086532 32188 raft_consensus.cc:729] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.086793 32188 consensus_queue.cc:260] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.087438 32188 ts_tablet_manager.cc:1398] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:58:50.087818 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.088744 32192 tablet_bootstrap.cc:492] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:50.088835 32192 ts_tablet_manager.cc:1367] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:58:50.089902 32192 raft_consensus.cc:348] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.090610 32192 raft_consensus.cc:374] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.090672 32192 raft_consensus.cc:729] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.090785 32192 consensus_queue.cc:260] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.091589 32192 ts_tablet_manager.cc:1398] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240502 01:58:50.095499 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.101621 31227 raft_consensus.cc:482] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:50.101712 31227 raft_consensus.cc:504] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.102146 31227 leader_election.cc:288] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.102425 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1384cb0374899a6e17a550e1f7cba" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:50.102535 27578 raft_consensus.cc:2454] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:50.103032 27113 leader_election.cc:302] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:50.103406 32084 raft_consensus.cc:2781] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:50.103475 32084 raft_consensus.cc:482] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:50.103525 32084 raft_consensus.cc:3037] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.104631 32084 raft_consensus.cc:504] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.104998 32084 leader_election.cc:288] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.105226 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1384cb0374899a6e17a550e1f7cba" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:50.105327 27578 raft_consensus.cc:3037] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:50.105499 27110 leader_election.cc:334] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.106269 27578 raft_consensus.cc:2454] T 7db1384cb0374899a6e17a550e1f7cba P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:50.106566 27113 leader_election.cc:302] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:50.106756 31227 raft_consensus.cc:2781] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:50.106861 31227 raft_consensus.cc:686] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:50.107002 31227 consensus_queue.cc:237] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
W20240502 01:58:50.107743 27110 leader_election.cc:334] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.108991 24479 catalog_manager.cc:5484] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.115657 24632 consensus_queue.cc:237] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:50.117336 25031 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:50.117636 27578 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:50.117962 32232 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:58:50.118216 24567 consensus_peers.cc:485] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Couldn't send request to peer 92cb785c165749eca49af5784756461c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cccdcec2dc2441b29fe7c7f261478bf1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.118248 31768 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:50.119086 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:50.120158 31768 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:50.120585 25031 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:50.121845 27578 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } } }
I20240502 01:58:50.123528 24479 catalog_manager.cc:5484] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 2, NON_VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.125480 27557 tablet_service.cc:1467] Processing CreateTablet for tablet b74364edd8a444f7b43a22bc8087548c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e5a13126ab14ed38a4e4f7a7c9af4b3]), partition=
I20240502 01:58:50.125525 27423 tablet_service.cc:1467] Processing CreateTablet for tablet b74364edd8a444f7b43a22bc8087548c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e5a13126ab14ed38a4e4f7a7c9af4b3]), partition=
I20240502 01:58:50.125705 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74364edd8a444f7b43a22bc8087548c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.125713 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74364edd8a444f7b43a22bc8087548c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.127807 32191 tablet_bootstrap.cc:492] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:50.128818 32191 tablet_bootstrap.cc:654] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.128911 32192 tablet_bootstrap.cc:492] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:50.129766 32192 tablet_bootstrap.cc:654] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.130800 32191 tablet_bootstrap.cc:492] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:50.130908 32191 ts_tablet_manager.cc:1367] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:50.131431 32191 raft_consensus.cc:348] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.131533 32191 raft_consensus.cc:374] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.131557 32191 raft_consensus.cc:729] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.131652 32192 tablet_bootstrap.cc:492] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:50.131668 32191 consensus_queue.cc:260] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.131709 32192 ts_tablet_manager.cc:1367] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:50.131933 32191 ts_tablet_manager.cc:1398] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:50.132076 32192 raft_consensus.cc:348] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.132161 32192 raft_consensus.cc:374] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.132198 32192 raft_consensus.cc:729] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.132302 32192 consensus_queue.cc:260] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.132572 32192 ts_tablet_manager.cc:1398] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:58:50.134269 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.144176 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.153970 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.172055 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.173436 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.180765 24765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf"
dest_uuid: "92cb785c165749eca49af5784756461c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60616
I20240502 01:58:50.180872 24765 raft_consensus.cc:595] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Received request to transfer leadership
I20240502 01:58:50.187162 31227 raft_consensus.cc:1050] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:50.187444 32084 raft_consensus.cc:482] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:50.187528 32084 raft_consensus.cc:504] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.187626 31227 consensus_queue.cc:237] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:50.187902 32084 leader_election.cc:288] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.188571 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252502ad2e124996b345888706e674d5" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:50.188706 27445 raft_consensus.cc:2454] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:50.189114 27113 leader_election.cc:302] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
W20240502 01:58:50.189203 27110 leader_election.cc:334] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.189247 27445 raft_consensus.cc:1261] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.189280 29098 raft_consensus.cc:2781] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:50.189330 29098 raft_consensus.cc:482] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:50.189371 29098 raft_consensus.cc:3037] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.189528 24897 raft_consensus.cc:1261] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:50.189603 30520 consensus_queue.cc:1035] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:50.189879 32084 consensus_queue.cc:1035] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:50.190395 29098 raft_consensus.cc:504] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.190725 29098 leader_election.cc:288] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.190867 31814 raft_consensus.cc:482] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:50.190966 31814 raft_consensus.cc:504] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.191357 31814 leader_election.cc:288] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
W20240502 01:58:50.191725 27110 leader_election.cc:334] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.191784 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28b2f0e50da4719aabefb28c34c2a0a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:50.191911 27177 raft_consensus.cc:2454] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:50.192114 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28b2f0e50da4719aabefb28c34c2a0a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:50.192219 27577 raft_consensus.cc:2454] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:50.192725 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252502ad2e124996b345888706e674d5" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:50.192818 27446 raft_consensus.cc:3037] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.193778 24832 leader_election.cc:302] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:50.193835 27446 raft_consensus.cc:2454] T 252502ad2e124996b345888706e674d5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:50.194255 27113 leader_election.cc:302] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:50.194813 32084 raft_consensus.cc:2781] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:50.194888 32084 raft_consensus.cc:686] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:50.194960 31814 raft_consensus.cc:2781] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:50.195012 31814 raft_consensus.cc:482] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:50.195044 31814 raft_consensus.cc:3037] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.195013 32084 consensus_queue.cc:237] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.195955 31814 raft_consensus.cc:504] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.196241 31814 leader_election.cc:288] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:50.196979 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28b2f0e50da4719aabefb28c34c2a0a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:50.197077 27177 raft_consensus.cc:3037] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.197508 24482 catalog_manager.cc:5484] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.198035 27177 raft_consensus.cc:2454] T e28b2f0e50da4719aabefb28c34c2a0a P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:50.198139 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28b2f0e50da4719aabefb28c34c2a0a" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:50.198251 27577 raft_consensus.cc:3037] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.198570 24832 leader_election.cc:302] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:50.198755 31814 raft_consensus.cc:2781] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:50.198884 31814 raft_consensus.cc:686] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:50.199010 31814 consensus_queue.cc:237] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:50.199172 27577 raft_consensus.cc:2454] T e28b2f0e50da4719aabefb28c34c2a0a P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:50.199482 29098 raft_consensus.cc:2932] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.201087 24897 raft_consensus.cc:2932] T 8919df9a8a574882bad4995fc5df7e50 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:50.201349 27445 raft_consensus.cc:2932] T 8919df9a8a574882bad4995fc5df7e50 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
W20240502 01:58:50.202306 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.202689 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.203065 24483 catalog_manager.cc:5484] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.203759 24482 catalog_manager.cc:5484] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:50.204941 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.210404 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.222487 24632 consensus_queue.cc:237] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.223686 27577 raft_consensus.cc:1261] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:58:50.223758 24567 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): Couldn't send request to peer f46bcd2bc124471981349b1e2726253b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 334184346780410a9f7efe298ae4d5e1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.224026 32232 consensus_queue.cc:1035] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:58:50.224205 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.225734 32232 raft_consensus.cc:2932] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:50.226100 27577 raft_consensus.cc:2932] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } } }
I20240502 01:58:50.228489 24482 catalog_manager.cc:5484] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index -1 to 2, NON_VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) added. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.263585 24765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf"
dest_uuid: "92cb785c165749eca49af5784756461c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60648
I20240502 01:58:50.263731 24765 raft_consensus.cc:595] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Received request to transfer leadership
I20240502 01:58:50.339448 31949 raft_consensus.cc:482] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:50.339569 31949 raft_consensus.cc:504] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.339926 31949 leader_election.cc:288] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:50.340301 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74364edd8a444f7b43a22bc8087548c" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:50.340456 27445 raft_consensus.cc:2454] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:50.340744 27514 leader_election.cc:302] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
W20240502 01:58:50.340842 27510 leader_election.cc:334] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.340904 31949 raft_consensus.cc:2781] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:50.340956 31949 raft_consensus.cc:482] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:50.340984 31949 raft_consensus.cc:3037] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.341990 31949 raft_consensus.cc:504] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.342321 31949 leader_election.cc:288] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:50.342590 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74364edd8a444f7b43a22bc8087548c" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:50.342700 27445 raft_consensus.cc:3037] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:50.343457 27445 raft_consensus.cc:2454] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:50.343714 27514 leader_election.cc:302] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:50.343854 31949 raft_consensus.cc:2781] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:50.343923 31949 raft_consensus.cc:686] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:50.344053 31949 consensus_queue.cc:237] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
W20240502 01:58:50.344153 27510 leader_election.cc:334] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.346105 24481 catalog_manager.cc:5484] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.449790 32183 raft_consensus.cc:979] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: : Instructing follower 1cc0a6338e0a467dbdc6efb3e67a6a09 to start an election
I20240502 01:58:50.449898 32183 raft_consensus.cc:1067] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Signalling peer 1cc0a6338e0a467dbdc6efb3e67a6a09 to start an election
I20240502 01:58:50.450340 27445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf"
dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
from {username='slave'} at 127.17.243.2:33651
I20240502 01:58:50.450479 27445 raft_consensus.cc:482] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:50.450539 27445 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:50.451335 27445 raft_consensus.cc:504] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:50.451730 27445 leader_election.cc:288] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 3 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:50.452018 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:50.452044 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:50.452165 24765 raft_consensus.cc:3032] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:58:50.452170 24632 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:50.452211 24765 raft_consensus.cc:729] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:50.452211 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44be529dec974eac8419efe9c19b30bf" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:50.452322 27577 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:50.452349 24765 consensus_queue.cc:260] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:50.452533 24765 raft_consensus.cc:3037] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:50.452903 24632 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 3.
I20240502 01:58:50.453271 27577 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 3.
I20240502 01:58:50.453465 24765 raft_consensus.cc:2454] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 3.
I20240502 01:58:50.453570 27380 leader_election.cc:302] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:50.453724 32105 raft_consensus.cc:2781] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:58:50.453817 32105 raft_consensus.cc:686] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 3 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:50.453990 32105 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:50.455865 24481 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 2 to 3, leader changed from 92cb785c165749eca49af5784756461c (127.17.243.2) to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 3 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:50.463853 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240502 01:58:50.468236 27510 consensus_peers.cc:485] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.468238 27445 raft_consensus.cc:1261] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:50.468595 31949 consensus_queue.cc:1035] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:50.471860 32364 tablet.cc:1722] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:50.472204 32364 tablet.cc:1620] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:50.473212 24482 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet b74364edd8a444f7b43a22bc8087548c (table rebalancer_dynamic_table [id=5e5a13126ab14ed38a4e4f7a7c9af4b3]). Expected version 1 got 0
I20240502 01:58:50.475304 32365 tablet.cc:1722] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:50.475634 32365 tablet.cc:1620] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:50.476102 24483 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet b74364edd8a444f7b43a22bc8087548c (table rebalancer_dynamic_table [id=5e5a13126ab14ed38a4e4f7a7c9af4b3]). Expected version 1 got 0
I20240502 01:58:50.478725 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5e5a13126ab14ed38a4e4f7a7c9af4b3] alter complete (version 1)
I20240502 01:58:50.506114 32391 ts_tablet_manager.cc:902] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:50.506642 32391 tablet_copy_client.cc:308] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:50.507186 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet cccdcec2dc2441b29fe7c7f261478bf1 from peer 92cb785c165749eca49af5784756461c ({username='slave'} at 127.17.243.2:49771)
I20240502 01:58:50.507300 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet cccdcec2dc2441b29fe7c7f261478bf1 from peer 92cb785c165749eca49af5784756461c at {username='slave'} at 127.17.243.2:49771: session id = 92cb785c165749eca49af5784756461c-cccdcec2dc2441b29fe7c7f261478bf1
I20240502 01:58:50.508201 24652 tablet_copy_source_session.cc:205] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:50.509053 32391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccdcec2dc2441b29fe7c7f261478bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.511508 32391 tablet_copy_client.cc:783] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:50.511694 32391 tablet_copy_client.cc:647] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:50.512923 32391 tablet_copy_client.cc:523] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:50.514094 32391 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:50.518585 32391 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:50.518918 32391 tablet_bootstrap.cc:492] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: Bootstrap complete.
I20240502 01:58:50.519299 32391 ts_tablet_manager.cc:1367] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240502 01:58:50.519846 32391 raft_consensus.cc:348] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:50.520097 32391 raft_consensus.cc:729] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: LEARNER
I20240502 01:58:50.520272 32391 consensus_queue.cc:260] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: NON_VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: true } }
I20240502 01:58:50.520670 32391 ts_tablet_manager.cc:1398] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.521194 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session 92cb785c165749eca49af5784756461c-cccdcec2dc2441b29fe7c7f261478bf1 received from {username='slave'} at 127.17.243.2:49771
I20240502 01:58:50.521270 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session 92cb785c165749eca49af5784756461c-cccdcec2dc2441b29fe7c7f261478bf1 on tablet cccdcec2dc2441b29fe7c7f261478bf1 with peer 92cb785c165749eca49af5784756461c
I20240502 01:58:50.524291 27176 consensus_queue.cc:237] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } }
I20240502 01:58:50.525463 27445 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:50.525878 24765 raft_consensus.cc:1261] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:50.526072 30520 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:50.526343 32084 consensus_queue.cc:1035] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:50.528182 31227 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:50.528520 27445 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:50.529503 24765 raft_consensus.cc:2932] T 91d0d9525b5f4384861dc50d0bf4967d P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } } }
I20240502 01:58:50.530042 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91d0d9525b5f4384861dc50d0bf4967d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:50.531502 24483 catalog_manager.cc:5484] T 91d0d9525b5f4384861dc50d0bf4967d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 2 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:50.534016 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d0d9525b5f4384861dc50d0bf4967d with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:50.534379 32412 tablet_replica.cc:330] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:50.534533 32412 raft_consensus.cc:2229] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.534710 32412 raft_consensus.cc:2258] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.535274 32412 ts_tablet_manager.cc:1875] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:50.537266 32412 ts_tablet_manager.cc:1888] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:50.537339 32412 log.cc:1196] T 91d0d9525b5f4384861dc50d0bf4967d P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/91d0d9525b5f4384861dc50d0bf4967d
I20240502 01:58:50.537838 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 91d0d9525b5f4384861dc50d0bf4967d (table rebalance_test_table_15 [id=d674fa1f0b064c6f9b1f2415895f5018]) successfully deleted
I20240502 01:58:50.542317 24896 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:50.543557 27445 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:50.543623 24765 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:58:50.543803 24833 consensus_peers.cc:485] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ec1158a7414039b6f4bc855207c938. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.543994 32361 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:50.544171 31814 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:50.545872 32297 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:50.546305 24765 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:50.547550 27445 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:50.548406 24481 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.583029 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:50.587726 32364 tablet.cc:1722] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:50.588043 32364 tablet.cc:1620] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:50.588918 32365 tablet.cc:1722] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:50.589175 32365 tablet.cc:1620] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:50.593103 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5e5a13126ab14ed38a4e4f7a7c9af4b3] alter complete (version 2)
W20240502 01:58:50.608530 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:50.644191 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:50.654053 31949 raft_consensus.cc:482] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:50.654161 31949 raft_consensus.cc:504] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.654528 31949 leader_election.cc:288] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.655323 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
W20240502 01:58:50.655822 27510 leader_election.cc:334] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.655896 27510 leader_election.cc:302] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f, 9dd619a595e94004ac0f7f5a9054e7cc
I20240502 01:58:50.656029 31949 raft_consensus.cc:2726] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:50.657605 32084 consensus_queue.cc:1035] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:50.677395 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.679520 32084 consensus_queue.cc:1035] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:50.696354 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:50.700810 32364 tablet.cc:1722] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:50.701164 32365 tablet.cc:1722] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:50.705389 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=5e5a13126ab14ed38a4e4f7a7c9af4b3] alter complete (version 3)
I20240502 01:58:50.721256 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:50.724071 32297 consensus_queue.cc:1035] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:50.727283 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9ee234307a410ca578be50f798804d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=dd5a4a1469e9412a82b3941efeb51faa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.727331 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 6e608c8945504b18b1cd82d421f2d3b9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=dd5a4a1469e9412a82b3941efeb51faa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.727545 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9ee234307a410ca578be50f798804d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.727576 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e608c8945504b18b1cd82d421f2d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.728158 27423 tablet_service.cc:1467] Processing CreateTablet for tablet a2d5afdc9ff6458ca834de042f8e14ab (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=dd5a4a1469e9412a82b3941efeb51faa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.728385 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d5afdc9ff6458ca834de042f8e14ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.729308 32471 tablet_bootstrap.cc:492] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:50.729746 32472 tablet_bootstrap.cc:492] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:50.730484 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9ee234307a410ca578be50f798804d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=dd5a4a1469e9412a82b3941efeb51faa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.730576 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 6e608c8945504b18b1cd82d421f2d3b9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=dd5a4a1469e9412a82b3941efeb51faa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.730707 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9ee234307a410ca578be50f798804d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.730849 32472 tablet_bootstrap.cc:654] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.730927 32361 consensus_queue.cc:1035] T e28b2f0e50da4719aabefb28c34c2a0a P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:50.731215 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e608c8945504b18b1cd82d421f2d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.732585 32475 tablet_bootstrap.cc:492] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:50.732553 25012 tablet_service.cc:1467] Processing CreateTablet for tablet a2d5afdc9ff6458ca834de042f8e14ab (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=dd5a4a1469e9412a82b3941efeb51faa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:50.732785 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d5afdc9ff6458ca834de042f8e14ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.733587 32471 tablet_bootstrap.cc:654] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.733587 32475 tablet_bootstrap.cc:654] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.734760 32472 tablet_bootstrap.cc:492] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:50.734872 32472 ts_tablet_manager.cc:1367] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:50.735298 32472 raft_consensus.cc:348] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.735423 32472 raft_consensus.cc:374] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.735476 32472 raft_consensus.cc:729] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.735612 32472 consensus_queue.cc:260] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.735903 32472 ts_tablet_manager.cc:1398] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.736299 32475 tablet_bootstrap.cc:492] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:50.736397 32475 ts_tablet_manager.cc:1367] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:50.737112 32473 tablet_bootstrap.cc:492] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:50.737099 32475 raft_consensus.cc:348] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.737268 32475 raft_consensus.cc:374] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.737313 32479 tablet_bootstrap.cc:492] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:50.737340 32475 raft_consensus.cc:729] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.737547 32475 consensus_queue.cc:260] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.738809 32473 tablet_bootstrap.cc:654] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.739781 32479 tablet_bootstrap.cc:654] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.740710 32473 tablet_bootstrap.cc:492] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:50.740841 32473 ts_tablet_manager.cc:1367] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:50.741297 32473 raft_consensus.cc:348] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.741564 32473 raft_consensus.cc:374] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.741758 32473 raft_consensus.cc:729] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.741955 32473 consensus_queue.cc:260] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.742909 32473 ts_tablet_manager.cc:1398] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:50.743541 32471 tablet_bootstrap.cc:492] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:50.743623 32471 ts_tablet_manager.cc:1367] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:58:50.743680 32479 tablet_bootstrap.cc:492] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:50.743746 32479 ts_tablet_manager.cc:1367] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:50.744048 32471 raft_consensus.cc:348] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.744140 32471 raft_consensus.cc:374] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.744098 32479 raft_consensus.cc:348] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.744168 32471 raft_consensus.cc:729] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.744179 32479 raft_consensus.cc:374] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.744207 32479 raft_consensus.cc:729] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.744282 32471 consensus_queue.cc:260] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.744351 32479 consensus_queue.cc:260] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:50.744537 32471 ts_tablet_manager.cc:1398] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:50.745352 32479 ts_tablet_manager.cc:1398] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:50.746814 32482 ts_tablet_manager.cc:902] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: Initiating tablet copy from peer 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:50.747421 32482 tablet_copy_client.cc:308] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.1:36901
I20240502 01:58:50.747575 32475 ts_tablet_manager.cc:1398] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 01:58:50.747921 24652 tablet_copy_service.cc:140] P 588bfc090a5247598cd5d3cee5ad2007: Received BeginTabletCopySession request for tablet 334184346780410a9f7efe298ae4d5e1 from peer f46bcd2bc124471981349b1e2726253b ({username='slave'} at 127.17.243.3:36527)
I20240502 01:58:50.748023 24652 tablet_copy_service.cc:161] P 588bfc090a5247598cd5d3cee5ad2007: Beginning new tablet copy session on tablet 334184346780410a9f7efe298ae4d5e1 from peer f46bcd2bc124471981349b1e2726253b at {username='slave'} at 127.17.243.3:36527: session id = f46bcd2bc124471981349b1e2726253b-334184346780410a9f7efe298ae4d5e1
I20240502 01:58:50.748323 32475 tablet_bootstrap.cc:492] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:50.749696 24652 tablet_copy_source_session.cc:205] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:50.750468 32482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334184346780410a9f7efe298ae4d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.752488 32482 tablet_copy_client.cc:783] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:50.752769 32482 tablet_copy_client.cc:647] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:50.753773 32482 tablet_copy_client.cc:523] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:50.754962 32482 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:50.752826 32084 raft_consensus.cc:482] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:50.756213 32084 raft_consensus.cc:504] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.756531 32084 leader_election.cc:288] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.757019 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:50.757438 32475 tablet_bootstrap.cc:654] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:50.758201 27110 leader_election.cc:334] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.758239 32482 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:50.758262 27110 leader_election.cc:302] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc; no voters: 1049b4c8783b48f39be22bd113ad349f, bb653087e8554c7591cf547dd969f3be
I20240502 01:58:50.758440 32084 raft_consensus.cc:2726] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:50.758451 32482 tablet_bootstrap.cc:492] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: Bootstrap complete.
I20240502 01:58:50.758581 32482 ts_tablet_manager.cc:1367] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:58:50.758949 32482 raft_consensus.cc:348] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.759053 32482 raft_consensus.cc:729] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: LEARNER
I20240502 01:58:50.759145 32482 consensus_queue.cc:260] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.759498 32475 tablet_bootstrap.cc:492] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:50.759620 32475 ts_tablet_manager.cc:1367] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 01:58:50.760020 32482 ts_tablet_manager.cc:1398] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.760425 24652 tablet_copy_service.cc:342] P 588bfc090a5247598cd5d3cee5ad2007: Request end of tablet copy session f46bcd2bc124471981349b1e2726253b-334184346780410a9f7efe298ae4d5e1 received from {username='slave'} at 127.17.243.3:36527
I20240502 01:58:50.760502 24652 tablet_copy_service.cc:433] P 588bfc090a5247598cd5d3cee5ad2007: ending tablet copy session f46bcd2bc124471981349b1e2726253b-334184346780410a9f7efe298ae4d5e1 on tablet 334184346780410a9f7efe298ae4d5e1 with peer f46bcd2bc124471981349b1e2726253b
I20240502 01:58:50.760855 32475 raft_consensus.cc:348] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.760957 32475 raft_consensus.cc:374] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.761018 32475 raft_consensus.cc:729] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.761271 32475 consensus_queue.cc:260] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.761533 32475 ts_tablet_manager.cc:1398] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:50.802698 32084 raft_consensus.cc:482] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:50.802770 32084 raft_consensus.cc:504] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.803043 32084 leader_election.cc:288] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.803300 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9ee234307a410ca578be50f798804d" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:50.803400 24896 raft_consensus.cc:2454] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:50.803627 27113 leader_election.cc:302] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:50.803851 32084 raft_consensus.cc:2781] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:50.803906 32084 raft_consensus.cc:482] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:50.803938 32084 raft_consensus.cc:3037] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:50.804248 27110 leader_election.cc:334] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.804713 32084 raft_consensus.cc:504] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.804957 32084 leader_election.cc:288] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.805213 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9ee234307a410ca578be50f798804d" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:50.805305 24896 raft_consensus.cc:3037] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:50.805812 27110 leader_election.cc:334] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.806005 24896 raft_consensus.cc:2454] T 1d9ee234307a410ca578be50f798804d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:50.806281 27113 leader_election.cc:302] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:50.806416 32084 raft_consensus.cc:2781] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:50.806496 32084 raft_consensus.cc:686] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:50.806605 32084 consensus_queue.cc:237] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:50.808306 24481 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:50.808416 24481 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:50.810201 24481 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet b74364edd8a444f7b43a22bc8087548c
I20240502 01:58:50.810122 24482 catalog_manager.cc:5484] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.810693 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet b74364edd8a444f7b43a22bc8087548c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:50 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:50.810887 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet b74364edd8a444f7b43a22bc8087548c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:50 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:50.810976 32496 tablet_replica.cc:330] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:50.811116 32496 raft_consensus.cc:2229] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:50.811443 32496 raft_consensus.cc:2258] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.812031 32496 ts_tablet_manager.cc:1875] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:50.812126 32497 tablet_replica.cc:330] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:50.812280 32497 raft_consensus.cc:2229] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.812449 32497 raft_consensus.cc:2258] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.813520 32497 ts_tablet_manager.cc:1875] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:50.814694 32496 ts_tablet_manager.cc:1888] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:50.814811 32496 log.cc:1196] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/b74364edd8a444f7b43a22bc8087548c
I20240502 01:58:50.815148 32496 ts_tablet_manager.cc:1909] T b74364edd8a444f7b43a22bc8087548c P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:50.815568 32497 ts_tablet_manager.cc:1888] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:50.815672 32497 log.cc:1196] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/b74364edd8a444f7b43a22bc8087548c
I20240502 01:58:50.815825 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet b74364edd8a444f7b43a22bc8087548c (table rebalancer_dynamic_table_new_name [id=5e5a13126ab14ed38a4e4f7a7c9af4b3]) successfully deleted
I20240502 01:58:50.815945 32497 ts_tablet_manager.cc:1909] T b74364edd8a444f7b43a22bc8087548c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:50.816522 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet b74364edd8a444f7b43a22bc8087548c (table rebalancer_dynamic_table_new_name [id=5e5a13126ab14ed38a4e4f7a7c9af4b3]) successfully deleted
I20240502 01:58:50.847324 32100 raft_consensus.cc:482] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:50.847431 32100 raft_consensus.cc:504] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.847774 32100 leader_election.cc:288] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:50.848100 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:50.848237 27176 raft_consensus.cc:2454] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
W20240502 01:58:50.848662 27510 leader_election.cc:334] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.848795 27513 leader_election.cc:302] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:50.849046 32100 raft_consensus.cc:2781] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:50.849102 32100 raft_consensus.cc:482] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:50.849154 32100 raft_consensus.cc:3037] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:50.850127 32100 raft_consensus.cc:504] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.850458 32100 leader_election.cc:288] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:50.850736 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ae33b0741c48c090dda6d4a819ef02" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:50.850833 27176 raft_consensus.cc:3037] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:58:50.851603 27510 leader_election.cc:334] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.852041 27176 raft_consensus.cc:2454] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 2.
I20240502 01:58:50.852277 27513 leader_election.cc:302] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:50.852407 32100 raft_consensus.cc:2781] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:50.852486 32100 raft_consensus.cc:686] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 2 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:50.852613 32100 consensus_queue.cc:237] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } }
I20240502 01:58:50.854238 24482 catalog_manager.cc:5484] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 1 to 2, leader changed from 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 2 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:50.872701 24630 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 3 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:58:50.873059 31448 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:50.876890 24765 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 3 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:58:50.877221 31448 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:50.882671 27577 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 3 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:58:50.883252 28447 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:50.911998 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:50.916489 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 2566533bf5ce43d5be0b60a92819cb77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db]), partition=
I20240502 01:58:50.916591 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 2566533bf5ce43d5be0b60a92819cb77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db]), partition=
I20240502 01:58:50.916607 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 2566533bf5ce43d5be0b60a92819cb77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db]), partition=
I20240502 01:58:50.916734 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2566533bf5ce43d5be0b60a92819cb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.916781 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2566533bf5ce43d5be0b60a92819cb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.916827 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2566533bf5ce43d5be0b60a92819cb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:50.918932 32475 tablet_bootstrap.cc:492] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:50.919245 32536 tablet_bootstrap.cc:492] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:50.919840 32475 tablet_bootstrap.cc:654] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.920086 32536 tablet_bootstrap.cc:654] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.920881 32537 tablet_bootstrap.cc:492] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:50.921545 32475 tablet_bootstrap.cc:492] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:50.921629 32475 ts_tablet_manager.cc:1367] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:50.921706 32537 tablet_bootstrap.cc:654] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:50.922168 32475 raft_consensus.cc:348] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:50.922264 32475 raft_consensus.cc:374] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.922295 32475 raft_consensus.cc:729] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.922397 32475 consensus_queue.cc:260] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:50.922621 32536 tablet_bootstrap.cc:492] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:50.922690 32536 ts_tablet_manager.cc:1367] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:50.922685 32475 ts_tablet_manager.cc:1398] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.922758 32537 tablet_bootstrap.cc:492] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:50.922830 32537 ts_tablet_manager.cc:1367] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:50.923089 32536 raft_consensus.cc:348] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:50.923175 32536 raft_consensus.cc:374] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.923199 32536 raft_consensus.cc:729] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.923220 32537 raft_consensus.cc:348] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:50.923314 32537 raft_consensus.cc:374] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:50.923293 32536 consensus_queue.cc:260] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:50.923342 32537 raft_consensus.cc:729] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:50.923445 32537 consensus_queue.cc:260] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:50.923527 32536 ts_tablet_manager.cc:1398] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.923703 32537 ts_tablet_manager.cc:1398] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:50.954977 32100 raft_consensus.cc:482] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:50.955057 32100 raft_consensus.cc:504] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.955392 32100 leader_election.cc:288] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:50.955662 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e608c8945504b18b1cd82d421f2d3b9" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:50.955789 27176 raft_consensus.cc:2454] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:50.956645 27513 leader_election.cc:302] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:50.957317 31949 raft_consensus.cc:2781] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:50.957473 31949 raft_consensus.cc:482] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:50.957597 31949 raft_consensus.cc:3037] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:50.957649 27510 leader_election.cc:334] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.959437 32147 raft_consensus.cc:482] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:50.959612 31949 raft_consensus.cc:504] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.959590 32147 raft_consensus.cc:504] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:50.959949 31949 leader_election.cc:288] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:50.960002 32147 leader_election.cc:288] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:50.960399 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e608c8945504b18b1cd82d421f2d3b9" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:50.960362 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:50.960492 27176 raft_consensus.cc:3037] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:50.960974 27510 leader_election.cc:334] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:58:50.961253 24698 leader_election.cc:334] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:50.961330 24698 leader_election.cc:302] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92cb785c165749eca49af5784756461c; no voters: 1049b4c8783b48f39be22bd113ad349f, bb653087e8554c7591cf547dd969f3be
I20240502 01:58:50.961341 27176 raft_consensus.cc:2454] T 6e608c8945504b18b1cd82d421f2d3b9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:50.961474 32147 raft_consensus.cc:2726] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:50.961584 27513 leader_election.cc:302] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:50.961719 31949 raft_consensus.cc:2781] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:50.961796 31949 raft_consensus.cc:686] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:50.961894 31949 consensus_queue.cc:237] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:50.963541 24483 catalog_manager.cc:5484] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.021075 32147 raft_consensus.cc:482] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:51.021153 32147 raft_consensus.cc:504] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.021441 32147 leader_election.cc:288] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:51.021751 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2566533bf5ce43d5be0b60a92819cb77" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:51.021813 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2566533bf5ce43d5be0b60a92819cb77" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:51.021868 27176 raft_consensus.cc:2454] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:51.021965 24630 raft_consensus.cc:2454] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:51.022168 24700 leader_election.cc:302] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:51.022367 32147 raft_consensus.cc:2781] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:51.022471 32147 raft_consensus.cc:482] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:51.022503 32147 raft_consensus.cc:3037] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.023612 32147 raft_consensus.cc:504] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
W20240502 01:58:51.023795 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet 1d9ee234307a410ca578be50f798804d: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.023980 32147 leader_election.cc:288] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:51.024535 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2566533bf5ce43d5be0b60a92819cb77" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:51.024660 27176 raft_consensus.cc:3037] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.024646 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2566533bf5ce43d5be0b60a92819cb77" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:51.024744 24630 raft_consensus.cc:3037] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.025437 27176 raft_consensus.cc:2454] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:51.025518 24630 raft_consensus.cc:2454] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:51.025700 24700 leader_election.cc:302] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:51.025877 32147 raft_consensus.cc:2781] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:51.025987 32147 raft_consensus.cc:686] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:51.026111 32147 consensus_queue.cc:237] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.027833 24483 catalog_manager.cc:5484] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.034765 32467 raft_consensus.cc:482] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:51.034865 32467 raft_consensus.cc:504] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.035187 32467 leader_election.cc:288] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:51.035471 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d5afdc9ff6458ca834de042f8e14ab" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:51.035571 27445 raft_consensus.cc:2454] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:51.035810 24966 leader_election.cc:302] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:51.035940 32467 raft_consensus.cc:2781] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:51.035993 32467 raft_consensus.cc:482] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:51.036034 32467 raft_consensus.cc:3037] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:51.036083 24963 leader_election.cc:334] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.036875 32467 raft_consensus.cc:504] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.037163 32467 leader_election.cc:288] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:51.037456 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d5afdc9ff6458ca834de042f8e14ab" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:51.037544 27445 raft_consensus.cc:3037] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:51.038165 24963 leader_election.cc:334] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.038498 27445 raft_consensus.cc:2454] T a2d5afdc9ff6458ca834de042f8e14ab P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:51.038751 24966 leader_election.cc:302] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:51.038986 31921 raft_consensus.cc:2781] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:51.039125 31921 raft_consensus.cc:686] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:51.039265 31921 consensus_queue.cc:237] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.040966 24481 catalog_manager.cc:5484] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:51.081816 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.103888 24765 raft_consensus.cc:1203] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:51.122491 32590 ts_tablet_manager.cc:902] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:51.122893 32590 tablet_copy_client.cc:308] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:51.123312 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet d5ec1158a7414039b6f4bc855207c938 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:41115)
I20240502 01:58:51.123397 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet d5ec1158a7414039b6f4bc855207c938 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:41115: session id = bb653087e8554c7591cf547dd969f3be-d5ec1158a7414039b6f4bc855207c938
I20240502 01:58:51.124107 24918 tablet_copy_source_session.cc:205] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:51.124913 32590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ec1158a7414039b6f4bc855207c938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.126787 32590 tablet_copy_client.cc:783] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:51.126981 32590 tablet_copy_client.cc:647] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:51.128067 32590 tablet_copy_client.cc:523] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:51.129096 32590 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:51.134323 31949 raft_consensus.cc:482] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:51.134404 31949 raft_consensus.cc:504] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:51.134723 31949 leader_election.cc:288] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:51.135458 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:51.135581 24765 raft_consensus.cc:2454] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:51.136018 27514 leader_election.cc:302] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:51.136173 31949 raft_consensus.cc:2781] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:51.136215 31949 raft_consensus.cc:482] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:51.136250 31949 raft_consensus.cc:3037] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:58:51.136456 27510 leader_election.cc:334] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.137022 31949 raft_consensus.cc:504] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:51.137364 31949 leader_election.cc:288] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:51.138177 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2078a814ae1648e2bde5e521ef45af6d" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:51.138294 24765 raft_consensus.cc:3037] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:51.139204 24765 raft_consensus.cc:2454] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 2.
I20240502 01:58:51.139904 27514 leader_election.cc:302] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
W20240502 01:58:51.140002 27510 leader_election.cc:334] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.140180 31949 raft_consensus.cc:2781] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:51.140259 31949 raft_consensus.cc:686] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [term 2 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:51.140379 31949 consensus_queue.cc:237] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:51.141327 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:51.142294 24481 catalog_manager.cc:5484] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 1 to 2, leader changed from 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 2 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:51.145357 24630 raft_consensus.cc:1261] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:51.145499 27176 raft_consensus.cc:1261] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:51.145772 32183 consensus_queue.cc:1035] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:51.146093 32147 consensus_queue.cc:1035] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:51.148332 24483 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 2566533bf5ce43d5be0b60a92819cb77 (table rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db]). Expected version 1 got 0
I20240502 01:58:51.148625 24481 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 2566533bf5ce43d5be0b60a92819cb77 (table rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db]). Expected version 1 got 0
I20240502 01:58:51.149991 32600 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:51.150347 32600 tablet.cc:1620] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:51.150780 32601 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:51.150825 32534 mvcc.cc:205] Tried to move back new op lower bound from 7023063577167536128 to 7023063576684068864. Current Snapshot: MvccSnapshot[applied={T|T < 7023063577167536128}]
I20240502 01:58:51.152818 32590 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:51.153206 32590 tablet_bootstrap.cc:492] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:51.153420 32590 ts_tablet_manager.cc:1367] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.024s user 0.025s sys 0.000s
I20240502 01:58:51.153486 32601 tablet.cc:1620] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:51.153861 32590 raft_consensus.cc:348] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:51.154045 32590 raft_consensus.cc:729] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:51.154161 32590 consensus_queue.cc:260] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:51.154294 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db] alter complete (version 1)
I20240502 01:58:51.155560 32590 ts_tablet_manager.cc:1398] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:51.155889 24481 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 2566533bf5ce43d5be0b60a92819cb77 (table rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db]). Expected version 1 got 0
I20240502 01:58:51.156424 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-d5ec1158a7414039b6f4bc855207c938 received from {username='slave'} at 127.17.243.8:41115
I20240502 01:58:51.156497 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session bb653087e8554c7591cf547dd969f3be-d5ec1158a7414039b6f4bc855207c938 on tablet d5ec1158a7414039b6f4bc855207c938 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:51.158789 32602 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:51.159114 32602 tablet.cc:1620] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:51.167461 30520 raft_consensus.cc:482] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:51.167537 30520 raft_consensus.cc:504] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.167836 30520 leader_election.cc:288] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:51.168093 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:51.168210 27578 raft_consensus.cc:2454] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:51.168473 27113 leader_election.cc:302] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:51.168637 30520 raft_consensus.cc:2781] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:51.168673 30520 raft_consensus.cc:482] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1049b4c8783b48f39be22bd113ad349f)
I20240502 01:58:51.168696 30520 raft_consensus.cc:3037] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:58:51.168717 27110 leader_election.cc:334] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.169317 30520 raft_consensus.cc:504] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.169596 30520 leader_election.cc:288] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:51.169858 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37610aa1329476188c0cd8b92973ca6" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:51.169986 27578 raft_consensus.cc:3037] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:58:51.170437 27110 leader_election.cc:334] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.170706 27578 raft_consensus.cc:2454] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 2.
I20240502 01:58:51.170953 27113 leader_election.cc:302] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:51.171865 30520 raft_consensus.cc:2781] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:51.171965 30520 raft_consensus.cc:686] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:51.172081 30520 consensus_queue.cc:237] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.173566 24483 catalog_manager.cc:5484] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 1 to 2, leader changed from 1049b4c8783b48f39be22bd113ad349f (127.17.243.6) to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 2 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:51.194481 30520 consensus_queue.cc:1035] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:51.198503 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.215353 24896 raft_consensus.cc:1203] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:58:51.259018 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:51.263325 32600 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:51.263603 32600 tablet.cc:1620] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:51.263676 32602 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:51.263868 32601 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:51.263932 32602 tablet.cc:1620] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:51.264147 32601 tablet.cc:1620] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:51.269273 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=36b2f46e94ff4a5d8af8252cb4c039db] alter complete (version 2)
I20240502 01:58:51.281250 27176 raft_consensus.cc:1261] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:51.281603 31949 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:51.287768 24896 raft_consensus.cc:3037] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Advancing to term 2
I20240502 01:58:51.288823 24896 raft_consensus.cc:1261] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 2 LEARNER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:58:51.289258 32100 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: NON_VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:58:51.290256 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.292162 31949 raft_consensus.cc:1050] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:51.292515 31949 consensus_queue.cc:237] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:51.293962 27176 raft_consensus.cc:1261] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:58:51.293991 24896 raft_consensus.cc:1261] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 2 LEARNER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240502 01:58:51.294791 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.294903 31949 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:51.295130 31949 consensus_queue.cc:1035] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:51.298414 32100 raft_consensus.cc:2932] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.300871 24896 raft_consensus.cc:2932] T 84ae33b0741c48c090dda6d4a819ef02 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.302107 27176 raft_consensus.cc:2932] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.305176 24483 catalog_manager.cc:5484] T 84ae33b0741c48c090dda6d4a819ef02 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: config changed from index 2 to 4, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.333869 27445 consensus_queue.cc:237] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } }
I20240502 01:58:51.335171 27578 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 3 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:58:51.335247 24632 raft_consensus.cc:1261] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 3 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:58:51.335619 28447 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:51.335927 28447 consensus_queue.cc:1035] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:51.337523 28447 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:51.339035 27578 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P bb653087e8554c7591cf547dd969f3be [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:51.339170 24632 raft_consensus.cc:2932] T 44be529dec974eac8419efe9c19b30bf P 588bfc090a5247598cd5d3cee5ad2007 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } }
I20240502 01:58:51.339797 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44be529dec974eac8419efe9c19b30bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:51.341271 24479 catalog_manager.cc:5484] T 44be529dec974eac8419efe9c19b30bf P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 9 to 11, VOTER 92cb785c165749eca49af5784756461c (127.17.243.2) evicted. New cstate: current_term: 3 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.343748 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 44be529dec974eac8419efe9c19b30bf with delete_type TABLET_DATA_TOMBSTONED (TS 92cb785c165749eca49af5784756461c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:51.344098 32647 tablet_replica.cc:330] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:51.344229 32647 raft_consensus.cc:2229] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:51.344331 32647 raft_consensus.cc:2258] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:51.344867 32647 ts_tablet_manager.cc:1875] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:51.346989 32647 ts_tablet_manager.cc:1888] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 01:58:51.347062 32647 log.cc:1196] T 44be529dec974eac8419efe9c19b30bf P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/44be529dec974eac8419efe9c19b30bf
I20240502 01:58:51.347573 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 44be529dec974eac8419efe9c19b30bf (table rebalance_test_table_1 [id=c4b486e329a64e5a945a97f7a4db8a36]) successfully deleted
I20240502 01:58:51.371012 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:51.375867 32600 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:51.376442 32601 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:51.379603 32602 tablet.cc:1722] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:51.381152 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=36b2f46e94ff4a5d8af8252cb4c039db] alter complete (version 3)
I20240502 01:58:51.483809 24483 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:51.483929 24483 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:51.485859 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 2566533bf5ce43d5be0b60a92819cb77
I20240502 01:58:51.486444 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet 2566533bf5ce43d5be0b60a92819cb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:51 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:51.486449 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2566533bf5ce43d5be0b60a92819cb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:51 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:51.486593 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 2566533bf5ce43d5be0b60a92819cb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:51 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:51.486631 32647 tablet_replica.cc:330] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:51.486763 32647 raft_consensus.cc:2229] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:51.486770 32689 tablet_replica.cc:330] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:51.486976 32689 raft_consensus.cc:2229] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:51.486979 32647 raft_consensus.cc:2258] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:51.487048 32690 tablet_replica.cc:330] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:51.487171 32689 raft_consensus.cc:2258] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:51.487201 32690 raft_consensus.cc:2229] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:51.487586 32690 raft_consensus.cc:2258] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:51.487615 32647 ts_tablet_manager.cc:1875] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:51.487893 32689 ts_tablet_manager.cc:1875] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:51.488152 32690 ts_tablet_manager.cc:1875] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:51.489749 32689 ts_tablet_manager.cc:1888] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:51.489813 32689 log.cc:1196] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/2566533bf5ce43d5be0b60a92819cb77
I20240502 01:58:51.490115 32689 ts_tablet_manager.cc:1909] T 2566533bf5ce43d5be0b60a92819cb77 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:51.490094 32647 ts_tablet_manager.cc:1888] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:51.490183 32647 log.cc:1196] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/2566533bf5ce43d5be0b60a92819cb77
I20240502 01:58:51.490474 32647 ts_tablet_manager.cc:1909] T 2566533bf5ce43d5be0b60a92819cb77 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:51.490497 32690 ts_tablet_manager.cc:1888] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:51.490556 32690 log.cc:1196] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/2566533bf5ce43d5be0b60a92819cb77
I20240502 01:58:51.490681 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 2566533bf5ce43d5be0b60a92819cb77 (table rebalancer_dynamic_table_new_name [id=36b2f46e94ff4a5d8af8252cb4c039db]) successfully deleted
I20240502 01:58:51.490854 32690 ts_tablet_manager.cc:1909] T 2566533bf5ce43d5be0b60a92819cb77 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:51.491832 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 2566533bf5ce43d5be0b60a92819cb77 (table rebalancer_dynamic_table_new_name [id=36b2f46e94ff4a5d8af8252cb4c039db]) successfully deleted
I20240502 01:58:51.492750 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 2566533bf5ce43d5be0b60a92819cb77 (table rebalancer_dynamic_table_new_name [id=36b2f46e94ff4a5d8af8252cb4c039db]) successfully deleted
W20240502 01:58:51.494123 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.502141 31949 consensus_queue.cc:1035] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:51.505239 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.587666 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:51.588260 32589 raft_consensus.cc:1050] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER 92cb785c165749eca49af5784756461c to VOTER
I20240502 01:58:51.588609 32589 consensus_queue.cc:237] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:51.590884 25032 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:51.591261 24763 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:51.591354 27578 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:51.591826 32588 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:51.592166 32588 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:51.592459 32588 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:51.593343 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 781ba9bbf19e413bb8981c48073f3360 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a8c123da08c4dccbe6ebc42dd48121c]), partition=
I20240502 01:58:51.593567 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 781ba9bbf19e413bb8981c48073f3360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.595723 32699 tablet_bootstrap.cc:492] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:51.595738 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 781ba9bbf19e413bb8981c48073f3360 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a8c123da08c4dccbe6ebc42dd48121c]), partition=
I20240502 01:58:51.595965 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 781ba9bbf19e413bb8981c48073f3360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.597549 32699 tablet_bootstrap.cc:654] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.598227 32701 tablet_bootstrap.cc:492] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:51.600193 32701 tablet_bootstrap.cc:654] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.600591 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:51.600649 31768 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.601716 32701 tablet_bootstrap.cc:492] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:51.601843 32701 ts_tablet_manager.cc:1367] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:51.602332 24763 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.602898 32701 raft_consensus.cc:348] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.603001 32701 raft_consensus.cc:374] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.603030 32701 raft_consensus.cc:729] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.603137 32701 consensus_queue.cc:260] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.604038 24481 catalog_manager.cc:5484] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.604722 32701 ts_tablet_manager.cc:1398] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:51.606072 27578 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.608471 25032 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 92cb785c165749eca49af5784756461c (127.17.243.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.609576 32699 tablet_bootstrap.cc:492] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:51.609666 32699 ts_tablet_manager.cc:1367] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:58:51.610163 32699 raft_consensus.cc:348] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.610281 32699 raft_consensus.cc:374] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.610317 32699 raft_consensus.cc:729] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.610446 32699 consensus_queue.cc:260] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.610743 32699 ts_tablet_manager.cc:1398] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:51.610885 24632 consensus_queue.cc:237] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } }
I20240502 01:58:51.612648 24763 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:51.613121 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 57e64ca6504646b8b77c5d36e32d5d6c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.613402 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e64ca6504646b8b77c5d36e32d5d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.614003 24610 tablet_service.cc:1467] Processing CreateTablet for tablet f4db089eb5dd4f9e8fa45c1190c591ef (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.614200 27423 tablet_service.cc:1467] Processing CreateTablet for tablet f4db089eb5dd4f9e8fa45c1190c591ef (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.614282 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4db089eb5dd4f9e8fa45c1190c591ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.614472 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4db089eb5dd4f9e8fa45c1190c591ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.615895 27555 tablet_service.cc:1467] Processing CreateTablet for tablet da60bdbf6831421e9f9abec53c6e1d35 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.616122 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da60bdbf6831421e9f9abec53c6e1d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.616153 24744 tablet_service.cc:1467] Processing CreateTablet for tablet da60bdbf6831421e9f9abec53c6e1d35 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.616379 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da60bdbf6831421e9f9abec53c6e1d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.613121 27557 tablet_service.cc:1467] Processing CreateTablet for tablet f4db089eb5dd4f9e8fa45c1190c591ef (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.617540 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4db089eb5dd4f9e8fa45c1190c591ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.617977 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 57e64ca6504646b8b77c5d36e32d5d6c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c1b46ec1e6c0479aad6ff62ab46b7b3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:51.618212 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e64ca6504646b8b77c5d36e32d5d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:51.618367 32699 tablet_bootstrap.cc:492] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:51.619207 32699 tablet_bootstrap.cc:654] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.620308 32693 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:51.622551 32588 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.624169 25032 raft_consensus.cc:1261] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:51.624920 24473 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cccdcec2dc2441b29fe7c7f261478bf1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:51.625088 32589 consensus_queue.cc:1035] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:51.625264 24482 catalog_manager.cc:5484] T cccdcec2dc2441b29fe7c7f261478bf1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.625650 24763 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.625862 32711 tablet_bootstrap.cc:492] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:51.626286 25032 raft_consensus.cc:2932] T cccdcec2dc2441b29fe7c7f261478bf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false } } }
I20240502 01:58:51.626870 32711 tablet_bootstrap.cc:654] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.627561 32715 tablet_bootstrap.cc:492] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
W20240502 01:58:51.628013 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.628638 32715 tablet_bootstrap.cc:654] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.629884 32713 tablet_bootstrap.cc:492] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:51.630358 32699 tablet_bootstrap.cc:492] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:51.630435 32699 ts_tablet_manager.cc:1367] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240502 01:58:51.630827 32713 tablet_bootstrap.cc:654] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.630853 32711 tablet_bootstrap.cc:492] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:51.630942 32711 ts_tablet_manager.cc:1367] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:51.630882 32699 raft_consensus.cc:348] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.630995 32699 raft_consensus.cc:374] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.631027 32699 raft_consensus.cc:729] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.631155 32699 consensus_queue.cc:260] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.631426 32699 ts_tablet_manager.cc:1398] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:51.632246 32711 raft_consensus.cc:348] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.632390 32711 raft_consensus.cc:374] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.632448 32711 raft_consensus.cc:729] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.633186 32711 consensus_queue.cc:260] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.633399 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet cccdcec2dc2441b29fe7c7f261478bf1 with delete_type TABLET_DATA_TOMBSTONED (TS bb653087e8554c7591cf547dd969f3be not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:51.633692 32719 tablet_replica.cc:330] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:51.633857 32711 ts_tablet_manager.cc:1398] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:51.635501 32719 raft_consensus.cc:2229] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:51.635702 32719 raft_consensus.cc:2258] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:51.635939 32715 tablet_bootstrap.cc:492] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:51.636036 32715 ts_tablet_manager.cc:1367] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:51.636579 32715 raft_consensus.cc:348] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.636693 32715 raft_consensus.cc:374] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.636729 32715 raft_consensus.cc:729] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.636871 32715 consensus_queue.cc:260] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.637317 32715 ts_tablet_manager.cc:1398] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:51.637627 32715 tablet_bootstrap.cc:492] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:51.638546 32715 tablet_bootstrap.cc:654] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.639590 32713 tablet_bootstrap.cc:492] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:51.639683 32713 ts_tablet_manager.cc:1367] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:51.639927 32715 tablet_bootstrap.cc:492] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:51.640002 32715 ts_tablet_manager.cc:1367] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:51.640143 32713 raft_consensus.cc:348] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.640273 32713 raft_consensus.cc:374] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.640321 32713 raft_consensus.cc:729] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.640441 32715 raft_consensus.cc:348] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.640460 32713 consensus_queue.cc:260] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.640545 32715 raft_consensus.cc:374] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.640753 32713 ts_tablet_manager.cc:1398] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:51.640992 32713 tablet_bootstrap.cc:492] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:51.641412 32719 ts_tablet_manager.cc:1875] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:51.642194 32713 tablet_bootstrap.cc:654] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.640583 32715 raft_consensus.cc:729] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.642491 32715 consensus_queue.cc:260] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.642999 32715 ts_tablet_manager.cc:1398] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:51.644384 32719 ts_tablet_manager.cc:1888] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:58:51.644572 32719 log.cc:1196] T cccdcec2dc2441b29fe7c7f261478bf1 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/cccdcec2dc2441b29fe7c7f261478bf1
I20240502 01:58:51.645015 32713 tablet_bootstrap.cc:492] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:51.645165 32713 ts_tablet_manager.cc:1367] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:51.645113 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet cccdcec2dc2441b29fe7c7f261478bf1 (table rebalancer_extra_table_10 [id=9b7717ee95964c0ea364781d5fbffcca]) successfully deleted
I20240502 01:58:51.647899 32713 raft_consensus.cc:348] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.648015 32713 raft_consensus.cc:374] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.648084 32713 raft_consensus.cc:729] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.648214 32713 consensus_queue.cc:260] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.648927 32713 ts_tablet_manager.cc:1398] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:51.649350 32713 tablet_bootstrap.cc:492] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:51.650334 32713 tablet_bootstrap.cc:654] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:51.651430 32713 tablet_bootstrap.cc:492] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:51.651512 32713 ts_tablet_manager.cc:1367] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:51.651911 32713 raft_consensus.cc:348] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.651990 32713 raft_consensus.cc:374] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:51.652011 32713 raft_consensus.cc:729] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:51.652107 32713 consensus_queue.cc:260] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.652367 32713 ts_tablet_manager.cc:1398] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240502 01:58:51.656770 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.659405 27578 raft_consensus.cc:1261] T b37610aa1329476188c0cd8b92973ca6 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:51.659785 32605 consensus_queue.cc:1035] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:58:51.661603 31921 consensus_queue.cc:1035] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:51.672152 27577 raft_consensus.cc:1203] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:51.673871 32100 raft_consensus.cc:482] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:51.673970 32100 raft_consensus.cc:504] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.674297 32100 leader_election.cc:288] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:51.674724 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e64ca6504646b8b77c5d36e32d5d6c" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:51.674837 27445 raft_consensus.cc:2454] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:51.675287 27514 leader_election.cc:302] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
W20240502 01:58:51.675371 27510 leader_election.cc:334] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.675457 32100 raft_consensus.cc:2781] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:51.675505 32100 raft_consensus.cc:482] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:51.675539 32100 raft_consensus.cc:3037] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.676379 32100 raft_consensus.cc:504] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.676693 32100 leader_election.cc:288] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:51.677378 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e64ca6504646b8b77c5d36e32d5d6c" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:51.677487 27445 raft_consensus.cc:3037] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.678412 27445 raft_consensus.cc:2454] T 57e64ca6504646b8b77c5d36e32d5d6c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
W20240502 01:58:51.678627 27510 leader_election.cc:334] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.678720 27514 leader_election.cc:302] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:51.678857 32100 raft_consensus.cc:2781] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:51.678943 32100 raft_consensus.cc:686] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:51.679049 32100 consensus_queue.cc:237] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:51.680764 24482 catalog_manager.cc:5484] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:51.726908 24763 raft_consensus.cc:1261] T 2078a814ae1648e2bde5e521ef45af6d P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:58:51.727265 32100 consensus_queue.cc:1035] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:58:51.730329 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.730787 31768 raft_consensus.cc:1050] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007: attempting to promote NON_VOTER f46bcd2bc124471981349b1e2726253b to VOTER
I20240502 01:58:51.731230 31768 consensus_queue.cc:237] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } }
I20240502 01:58:51.732990 27577 raft_consensus.cc:1261] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:51.733032 24896 raft_consensus.cc:1261] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b [term 1 LEARNER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:58:51.733465 32588 consensus_queue.cc:1035] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:58:51.733574 32693 consensus_queue.cc:1035] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:58:51.733654 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.735677 32693 raft_consensus.cc:2932] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.736110 24896 raft_consensus.cc:2932] T 334184346780410a9f7efe298ae4d5e1 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.738086 27577 raft_consensus.cc:2932] T 334184346780410a9f7efe298ae4d5e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } } }
I20240502 01:58:51.738821 24481 catalog_manager.cc:5484] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 2 to 3, f46bcd2bc124471981349b1e2726253b (127.17.243.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.788933 31949 raft_consensus.cc:482] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:51.789016 31949 raft_consensus.cc:504] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.789323 31949 leader_election.cc:288] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:51.789604 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4db089eb5dd4f9e8fa45c1190c591ef" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:51.789605 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4db089eb5dd4f9e8fa45c1190c591ef" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:51.789726 27445 raft_consensus.cc:2454] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:51.789726 24632 raft_consensus.cc:2454] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:51.789985 27514 leader_election.cc:302] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:51.790133 31949 raft_consensus.cc:2781] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:51.790187 31949 raft_consensus.cc:482] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:51.790230 31949 raft_consensus.cc:3037] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.790894 31949 raft_consensus.cc:504] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.791199 31949 leader_election.cc:288] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:51.791428 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4db089eb5dd4f9e8fa45c1190c591ef" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:51.791544 24632 raft_consensus.cc:3037] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.791535 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4db089eb5dd4f9e8fa45c1190c591ef" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:51.791646 27445 raft_consensus.cc:3037] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.792476 24632 raft_consensus.cc:2454] T f4db089eb5dd4f9e8fa45c1190c591ef P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:51.792490 27445 raft_consensus.cc:2454] T f4db089eb5dd4f9e8fa45c1190c591ef P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:51.792762 27513 leader_election.cc:302] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:51.792992 31949 raft_consensus.cc:2781] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:51.793073 31949 raft_consensus.cc:686] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:51.793188 31949 consensus_queue.cc:237] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:51.794922 24481 catalog_manager.cc:5484] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:51.835188 31921 raft_consensus.cc:482] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:51.835274 31921 raft_consensus.cc:504] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.835590 31921 leader_election.cc:288] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:51.835822 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781ba9bbf19e413bb8981c48073f3360" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:51.835949 24763 raft_consensus.cc:2454] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:51.836227 24966 leader_election.cc:302] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:51.836375 31921 raft_consensus.cc:2781] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 01:58:51.836433 24963 leader_election.cc:334] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.836444 31921 raft_consensus.cc:482] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:51.836524 31921 raft_consensus.cc:3037] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.837256 31921 raft_consensus.cc:504] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.837563 31921 leader_election.cc:288] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:51.837844 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781ba9bbf19e413bb8981c48073f3360" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:51.837976 24763 raft_consensus.cc:3037] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:51.838450 24963 leader_election.cc:334] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.838822 24763 raft_consensus.cc:2454] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:51.839072 24966 leader_election.cc:302] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:51.839347 31921 raft_consensus.cc:2781] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:51.839417 31921 raft_consensus.cc:686] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:51.839534 31921 consensus_queue.cc:237] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:51.841140 24481 catalog_manager.cc:5484] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:51.858274 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:51.893179 31949 raft_consensus.cc:482] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:51.893294 31949 raft_consensus.cc:504] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.893616 31949 leader_election.cc:288] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:51.893908 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da60bdbf6831421e9f9abec53c6e1d35" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:51.894057 24763 raft_consensus.cc:2454] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:51.894284 27514 leader_election.cc:302] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:51.894460 31949 raft_consensus.cc:2781] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 01:58:51.894501 27510 leader_election.cc:334] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.894513 31949 raft_consensus.cc:482] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:51.894589 31949 raft_consensus.cc:3037] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:51.895306 31949 raft_consensus.cc:504] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.895606 31949 leader_election.cc:288] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:51.895896 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da60bdbf6831421e9f9abec53c6e1d35" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:51.895995 24763 raft_consensus.cc:3037] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:51.896514 27510 leader_election.cc:334] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:51.896996 24763 raft_consensus.cc:2454] T da60bdbf6831421e9f9abec53c6e1d35 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:51.897269 27514 leader_election.cc:302] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:51.897423 31949 raft_consensus.cc:2781] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:51.897518 31949 raft_consensus.cc:686] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:51.897646 31949 consensus_queue.cc:237] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:51.899606 24481 catalog_manager.cc:5484] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:51.932474 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:51.954999 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:51.958308 24763 raft_consensus.cc:1261] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:51.958695 32740 consensus_queue.cc:1035] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:51.958813 24963 consensus_peers.cc:485] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:51.960951 24481 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 781ba9bbf19e413bb8981c48073f3360 (table rebalancer_dynamic_table [id=8a8c123da08c4dccbe6ebc42dd48121c]). Expected version 1 got 0
I20240502 01:58:51.961110 358 tablet.cc:1722] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:51.961390 358 tablet.cc:1620] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:51.962334 359 tablet.cc:1722] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:51.962639 359 tablet.cc:1620] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:51.963606 24482 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 781ba9bbf19e413bb8981c48073f3360 (table rebalancer_dynamic_table [id=8a8c123da08c4dccbe6ebc42dd48121c]). Expected version 1 got 0
I20240502 01:58:51.964268 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=8a8c123da08c4dccbe6ebc42dd48121c] alter complete (version 1)
I20240502 01:58:52.067003 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:52.071944 358 tablet.cc:1722] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:52.072254 358 tablet.cc:1620] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:52.072216 359 tablet.cc:1722] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:52.072490 359 tablet.cc:1620] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:52.073588 31949 consensus_queue.cc:1035] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:52.077968 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:52.078554 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=8a8c123da08c4dccbe6ebc42dd48121c] alter complete (version 2)
W20240502 01:58:52.153031 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.159539 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.164848 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.179828 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240502 01:58:52.186169 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:52.189170 358 tablet.cc:1722] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:52.189579 359 tablet.cc:1722] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:52.194082 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=8a8c123da08c4dccbe6ebc42dd48121c] alter complete (version 3)
W20240502 01:58:52.227579 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet da60bdbf6831421e9f9abec53c6e1d35: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:52.269712 391 raft_consensus.cc:1050] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:52.270112 391 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:52.271529 27445 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:52.271508 27577 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:52.271577 24763 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:52.271936 32750 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:52.272097 393 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:52.272293 394 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:52.274334 393 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:52.274812 27577 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:52.275072 27445 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:52.276121 24763 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:52.277786 24482 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:52.296000 24481 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:52.296104 24481 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:52.298116 24481 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 781ba9bbf19e413bb8981c48073f3360
I20240502 01:58:52.298599 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 781ba9bbf19e413bb8981c48073f3360 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:52 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:52.298792 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 781ba9bbf19e413bb8981c48073f3360 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:52 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:52.299301 398 tablet_replica.cc:330] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:52.299369 399 tablet_replica.cc:330] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:52.299471 398 raft_consensus.cc:2229] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:52.299639 398 raft_consensus.cc:2258] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:52.299585 399 raft_consensus.cc:2229] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:52.300823 398 ts_tablet_manager.cc:1875] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:52.300897 399 raft_consensus.cc:2258] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:52.301358 399 ts_tablet_manager.cc:1875] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:52.302573 31949 consensus_queue.cc:1035] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:52.303146 398 ts_tablet_manager.cc:1888] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:52.303241 398 log.cc:1196] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/781ba9bbf19e413bb8981c48073f3360
I20240502 01:58:52.303567 398 ts_tablet_manager.cc:1909] T 781ba9bbf19e413bb8981c48073f3360 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:52.303673 399 ts_tablet_manager.cc:1888] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:52.303768 399 log.cc:1196] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/781ba9bbf19e413bb8981c48073f3360
I20240502 01:58:52.304098 399 ts_tablet_manager.cc:1909] T 781ba9bbf19e413bb8981c48073f3360 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:52.304172 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 781ba9bbf19e413bb8981c48073f3360 (table rebalancer_dynamic_table_new_name [id=8a8c123da08c4dccbe6ebc42dd48121c]) successfully deleted
I20240502 01:58:52.304796 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 781ba9bbf19e413bb8981c48073f3360 (table rebalancer_dynamic_table_new_name [id=8a8c123da08c4dccbe6ebc42dd48121c]) successfully deleted
I20240502 01:58:52.306345 31949 consensus_queue.cc:1035] T f4db089eb5dd4f9e8fa45c1190c591ef P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:52.341398 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:52.368420 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.400126 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:52.405237 27156 tablet_service.cc:1467] Processing CreateTablet for tablet a35596116445411f857f52eadd963a16 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665]), partition=
I20240502 01:58:52.405267 27558 tablet_service.cc:1467] Processing CreateTablet for tablet a35596116445411f857f52eadd963a16 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665]), partition=
I20240502 01:58:52.405470 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35596116445411f857f52eadd963a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.405511 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35596116445411f857f52eadd963a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.407850 413 tablet_bootstrap.cc:492] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:52.407946 414 tablet_bootstrap.cc:492] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:52.408799 413 tablet_bootstrap.cc:654] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.408809 414 tablet_bootstrap.cc:654] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.409693 413 tablet_bootstrap.cc:492] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:52.409710 414 tablet_bootstrap.cc:492] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:52.409770 414 ts_tablet_manager.cc:1367] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:52.409771 413 ts_tablet_manager.cc:1367] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:52.410213 413 raft_consensus.cc:348] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:52.410324 413 raft_consensus.cc:374] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.410364 413 raft_consensus.cc:729] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.410470 413 consensus_queue.cc:260] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:52.410216 414 raft_consensus.cc:348] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:52.410583 414 raft_consensus.cc:374] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.410722 414 raft_consensus.cc:729] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.410748 413 ts_tablet_manager.cc:1398] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:52.410849 414 consensus_queue.cc:260] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:52.411767 414 ts_tablet_manager.cc:1398] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:52.415962 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:52.422186 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdd8196cd074924910b47ff7c7771bb (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.422435 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdd8196cd074924910b47ff7c7771bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.422583 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdd8196cd074924910b47ff7c7771bb (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.422843 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdd8196cd074924910b47ff7c7771bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.424351 413 tablet_bootstrap.cc:492] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:52.424593 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdd8196cd074924910b47ff7c7771bb (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.424841 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdd8196cd074924910b47ff7c7771bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.425102 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 6f457b933f4a41b2b5588e767d76e325 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.425179 413 tablet_bootstrap.cc:654] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.425297 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f457b933f4a41b2b5588e767d76e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.426529 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 6f457b933f4a41b2b5588e767d76e325 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.426762 413 tablet_bootstrap.cc:492] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:52.426882 413 ts_tablet_manager.cc:1367] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:52.427436 413 raft_consensus.cc:348] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.427600 413 raft_consensus.cc:374] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.427672 413 raft_consensus.cc:729] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.427894 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f457b933f4a41b2b5588e767d76e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.427853 413 consensus_queue.cc:260] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.428205 413 ts_tablet_manager.cc:1398] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:52.430042 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 0180449195154cb49ebccb8b604af4e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.430263 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0180449195154cb49ebccb8b604af4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.430375 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 0180449195154cb49ebccb8b604af4e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.430579 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0180449195154cb49ebccb8b604af4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.432299 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 6f457b933f4a41b2b5588e767d76e325 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=34cd8ae7151d4344bc91c9fd9d633498]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:52.432508 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f457b933f4a41b2b5588e767d76e325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:52.432638 429 tablet_bootstrap.cc:492] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:52.433699 427 tablet_bootstrap.cc:492] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:52.434545 429 tablet_bootstrap.cc:654] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.434689 413 tablet_bootstrap.cc:492] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:52.435405 413 tablet_bootstrap.cc:654] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.436000 427 tablet_bootstrap.cc:654] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:52.436436 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.436586 413 tablet_bootstrap.cc:492] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:52.436654 413 ts_tablet_manager.cc:1367] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:52.436973 429 tablet_bootstrap.cc:492] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:52.436959 413 raft_consensus.cc:348] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.437024 413 raft_consensus.cc:374] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.437052 413 raft_consensus.cc:729] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.437073 429 ts_tablet_manager.cc:1367] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:52.437175 413 consensus_queue.cc:260] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.437582 429 raft_consensus.cc:348] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.437747 429 raft_consensus.cc:374] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.437825 429 raft_consensus.cc:729] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.438166 413 ts_tablet_manager.cc:1398] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:52.438249 429 consensus_queue.cc:260] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.438550 429 ts_tablet_manager.cc:1398] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:52.439309 427 tablet_bootstrap.cc:492] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:52.439422 427 ts_tablet_manager.cc:1367] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:52.439850 427 raft_consensus.cc:348] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.439992 427 raft_consensus.cc:374] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.440037 427 raft_consensus.cc:729] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.440153 427 consensus_queue.cc:260] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.440431 427 ts_tablet_manager.cc:1398] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:52.441064 427 tablet_bootstrap.cc:492] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:52.441922 427 tablet_bootstrap.cc:654] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.443830 430 tablet_bootstrap.cc:492] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:52.444384 427 tablet_bootstrap.cc:492] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:52.444535 427 ts_tablet_manager.cc:1367] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:52.444710 430 tablet_bootstrap.cc:654] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.445250 427 raft_consensus.cc:348] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.445514 427 raft_consensus.cc:374] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.445571 427 raft_consensus.cc:729] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.445704 427 consensus_queue.cc:260] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.446274 430 tablet_bootstrap.cc:492] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:52.446357 430 ts_tablet_manager.cc:1367] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:52.446991 430 raft_consensus.cc:348] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.447141 430 raft_consensus.cc:374] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.447206 430 raft_consensus.cc:729] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.447350 430 consensus_queue.cc:260] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.447537 427 ts_tablet_manager.cc:1398] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:52.447652 430 ts_tablet_manager.cc:1398] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:52.447866 430 tablet_bootstrap.cc:492] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:52.448735 430 tablet_bootstrap.cc:654] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.456208 430 tablet_bootstrap.cc:492] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:52.456290 430 ts_tablet_manager.cc:1367] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:58:52.457142 430 raft_consensus.cc:348] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.457257 430 raft_consensus.cc:374] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.457319 430 raft_consensus.cc:729] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
W20240502 01:58:52.457527 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.457495 430 consensus_queue.cc:260] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.457808 430 ts_tablet_manager.cc:1398] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:52.458206 430 tablet_bootstrap.cc:492] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:52.458988 430 tablet_bootstrap.cc:654] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:52.460495 430 tablet_bootstrap.cc:492] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:52.460634 430 ts_tablet_manager.cc:1367] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:52.461019 430 raft_consensus.cc:348] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.461113 430 raft_consensus.cc:374] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:52.461138 430 raft_consensus.cc:729] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:52.461239 430 consensus_queue.cc:260] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.461488 430 ts_tablet_manager.cc:1398] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:52.464527 31448 raft_consensus.cc:482] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:52.464613 31448 raft_consensus.cc:504] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.464991 31448 leader_election.cc:288] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:52.465469 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd8196cd074924910b47ff7c7771bb" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:52.465600 24896 raft_consensus.cc:2454] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:52.465592 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd8196cd074924910b47ff7c7771bb" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:52.465711 27577 raft_consensus.cc:2454] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:52.466117 27380 leader_election.cc:302] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:52.466271 31448 raft_consensus.cc:2781] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:52.466337 31448 raft_consensus.cc:482] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:52.466388 31448 raft_consensus.cc:3037] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.467200 31448 raft_consensus.cc:504] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.467541 31448 leader_election.cc:288] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:52.467906 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd8196cd074924910b47ff7c7771bb" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:52.468006 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdd8196cd074924910b47ff7c7771bb" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:52.468035 27577 raft_consensus.cc:3037] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.468109 24896 raft_consensus.cc:3037] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.468982 27577 raft_consensus.cc:2454] T 2cdd8196cd074924910b47ff7c7771bb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:52.468982 24896 raft_consensus.cc:2454] T 2cdd8196cd074924910b47ff7c7771bb P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:52.469259 27380 leader_election.cc:302] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:52.469441 31448 raft_consensus.cc:2781] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:52.469547 31448 raft_consensus.cc:686] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:52.469676 31448 consensus_queue.cc:237] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.472045 24479 catalog_manager.cc:5484] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:52.483151 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50062
I20240502 01:58:52.483255 24896 raft_consensus.cc:595] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:52.498469 31949 consensus_queue.cc:1035] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:52.498714 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.521559 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.560431 32100 raft_consensus.cc:482] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:52.560520 32100 raft_consensus.cc:504] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:52.560858 32100 leader_election.cc:288] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:52.561131 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35596116445411f857f52eadd963a16" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:52.561252 27176 raft_consensus.cc:2454] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:52.561517 27513 leader_election.cc:302] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:52.561667 32100 raft_consensus.cc:2781] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:52.561720 32100 raft_consensus.cc:482] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:52.561759 32100 raft_consensus.cc:3037] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:52.561906 27510 leader_election.cc:334] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:52.562633 32100 raft_consensus.cc:504] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
I20240502 01:58:52.562943 32100 leader_election.cc:288] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:52.563433 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35596116445411f857f52eadd963a16" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:52.563534 27176 raft_consensus.cc:3037] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.564196 27176 raft_consensus.cc:2454] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:52.564432 27513 leader_election.cc:302] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:52.564577 32100 raft_consensus.cc:2781] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:52.564652 32100 raft_consensus.cc:686] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:52.564766 32100 consensus_queue.cc:237] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } }
W20240502 01:58:52.565814 27510 leader_election.cc:334] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:52.567114 24479 catalog_manager.cc:5484] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:52.567543 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.577507 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.593149 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.607120 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.623756 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:58:52.647145 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.665585 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.667780 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.679317 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.695139 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.701531 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:52.705725 27176 raft_consensus.cc:1261] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:52.706105 32100 consensus_queue.cc:1035] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:52.706509 27510 consensus_peers.cc:485] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:52.709057 24479 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet a35596116445411f857f52eadd963a16 (table rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665]). Expected version 1 got 0
I20240502 01:58:52.710767 479 tablet.cc:1722] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:52.711026 479 tablet.cc:1620] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:52.711172 24483 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet a35596116445411f857f52eadd963a16 (table rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665]). Expected version 1 got 0
I20240502 01:58:52.712074 480 tablet.cc:1722] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:52.712294 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet a35596116445411f857f52eadd963a16 (table rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665]). Expected version 1 got 0
I20240502 01:58:52.712358 480 tablet.cc:1620] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:52.713877 31448 raft_consensus.cc:482] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:52.714001 31448 raft_consensus.cc:504] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.714397 31448 leader_election.cc:288] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:52.715255 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0180449195154cb49ebccb8b604af4e2" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:52.715377 24896 raft_consensus.cc:2454] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:52.715638 27380 leader_election.cc:302] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
W20240502 01:58:52.715912 27377 leader_election.cc:334] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:52.716672 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665] alter complete (version 1)
I20240502 01:58:52.716951 31448 raft_consensus.cc:2781] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:52.717012 31448 raft_consensus.cc:482] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:52.717051 31448 raft_consensus.cc:3037] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.717918 31448 raft_consensus.cc:504] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.718304 31448 leader_election.cc:288] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707)
I20240502 01:58:52.719033 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0180449195154cb49ebccb8b604af4e2" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:52.719136 24896 raft_consensus.cc:3037] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:52.719858 27377 leader_election.cc:334] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:52.720078 24896 raft_consensus.cc:2454] T 0180449195154cb49ebccb8b604af4e2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:52.720389 27380 leader_election.cc:302] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters: 1049b4c8783b48f39be22bd113ad349f
I20240502 01:58:52.720541 31448 raft_consensus.cc:2781] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:52.720623 31448 raft_consensus.cc:686] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:52.720739 31448 consensus_queue.cc:237] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:52.722824 24479 catalog_manager.cc:5484] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1049b4c8783b48f39be22bd113ad349f" member_type: VOTER last_known_addr { host: "127.17.243.6" port: 41707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:52.728716 24896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50070
I20240502 01:58:52.728818 24896 raft_consensus.cc:595] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:52.737851 31921 raft_consensus.cc:482] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:52.737977 31921 raft_consensus.cc:504] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.738277 31921 leader_election.cc:288] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:52.738543 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f457b933f4a41b2b5588e767d76e325" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:52.738595 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f457b933f4a41b2b5588e767d76e325" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:52.738649 27445 raft_consensus.cc:2454] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:52.738710 27578 raft_consensus.cc:2454] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:52.738963 24966 leader_election.cc:302] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:52.739203 31921 raft_consensus.cc:2781] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:52.739254 31921 raft_consensus.cc:482] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:52.739291 31921 raft_consensus.cc:3037] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.740190 31921 raft_consensus.cc:504] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.740515 31921 leader_election.cc:288] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:52.740813 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f457b933f4a41b2b5588e767d76e325" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:52.740823 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f457b933f4a41b2b5588e767d76e325" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:52.740926 27578 raft_consensus.cc:3037] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.740913 27445 raft_consensus.cc:3037] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:52.741648 27445 raft_consensus.cc:2454] T 6f457b933f4a41b2b5588e767d76e325 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:52.741758 27578 raft_consensus.cc:2454] T 6f457b933f4a41b2b5588e767d76e325 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:52.742031 24966 leader_election.cc:302] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:52.742199 31921 raft_consensus.cc:2781] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:52.742288 31921 raft_consensus.cc:686] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:52.742401 31921 consensus_queue.cc:237] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:52.744305 24479 catalog_manager.cc:5484] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:52.758137 393 raft_consensus.cc:979] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: : Instructing follower 1cc0a6338e0a467dbdc6efb3e67a6a09 to start an election
I20240502 01:58:52.758209 393 raft_consensus.cc:1067] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Signalling peer 1cc0a6338e0a467dbdc6efb3e67a6a09 to start an election
I20240502 01:58:52.758574 27445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938"
dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
from {username='slave'} at 127.17.243.3:51335
I20240502 01:58:52.758661 27445 raft_consensus.cc:482] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:52.758706 27445 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:52.759510 27445 raft_consensus.cc:504] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:52.759847 27445 leader_election.cc:288] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 2 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:52.760073 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:52.760213 24896 raft_consensus.cc:3032] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:52.760262 24896 raft_consensus.cc:729] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:52.760308 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:52.760418 24763 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:52.760391 24896 consensus_queue.cc:260] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:52.760586 24896 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:52.761402 24763 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 2.
I20240502 01:58:52.761401 24896 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 2.
I20240502 01:58:52.761709 27380 leader_election.cc:302] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:52.761693 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ec1158a7414039b6f4bc855207c938" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:52.761802 27578 raft_consensus.cc:3037] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:52.761873 31448 raft_consensus.cc:2781] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:52.761976 31448 raft_consensus.cc:686] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:52.762121 31448 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:52.762712 27578 raft_consensus.cc:2454] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 2.
I20240502 01:58:52.764110 24479 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 1 to 2, leader changed from f46bcd2bc124471981349b1e2726253b (127.17.243.3) to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 2 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:52.764294 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:52.807747 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:52.818177 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:52.822325 479 tablet.cc:1722] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:52.822650 479 tablet.cc:1620] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:52.822743 480 tablet.cc:1722] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:52.823005 480 tablet.cc:1620] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:52.827445 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=b62f8c6911e54e648b977f60fa86c665] alter complete (version 2)
I20240502 01:58:52.930824 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:52.936601 479 tablet.cc:1722] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:52.937242 480 tablet.cc:1722] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:52.941942 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=b62f8c6911e54e648b977f60fa86c665] alter complete (version 3)
I20240502 01:58:53.016764 32105 consensus_queue.cc:1035] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.020016 531 consensus_queue.cc:1035] T 2cdd8196cd074924910b47ff7c7771bb P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:53.034950 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:53.045274 24483 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:53.045389 24483 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:53.047200 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet a35596116445411f857f52eadd963a16
I20240502 01:58:53.047658 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet a35596116445411f857f52eadd963a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:53 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:53.047755 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet a35596116445411f857f52eadd963a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:53 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:53.048043 535 tablet_replica.cc:330] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:53.048054 534 tablet_replica.cc:330] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:53.048184 535 raft_consensus.cc:2229] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:53.048184 534 raft_consensus.cc:2229] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:53.048306 534 raft_consensus.cc:2258] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:53.048384 535 raft_consensus.cc:2258] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:53.049010 535 ts_tablet_manager.cc:1875] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:53.049023 534 ts_tablet_manager.cc:1875] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:53.051157 535 ts_tablet_manager.cc:1888] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:53.051157 534 ts_tablet_manager.cc:1888] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:53.051236 534 log.cc:1196] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/a35596116445411f857f52eadd963a16
I20240502 01:58:53.051237 535 log.cc:1196] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/a35596116445411f857f52eadd963a16
I20240502 01:58:53.051501 535 ts_tablet_manager.cc:1909] T a35596116445411f857f52eadd963a16 P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:53.051501 534 ts_tablet_manager.cc:1909] T a35596116445411f857f52eadd963a16 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:53.052105 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet a35596116445411f857f52eadd963a16 (table rebalancer_dynamic_table_new_name [id=b62f8c6911e54e648b977f60fa86c665]) successfully deleted
I20240502 01:58:53.052186 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet a35596116445411f857f52eadd963a16 (table rebalancer_dynamic_table_new_name [id=b62f8c6911e54e648b977f60fa86c665]) successfully deleted
W20240502 01:58:53.144891 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:53.148929 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:53.153914 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 7c08ab332124429889f295eb70202100 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8]), partition=
I20240502 01:58:53.154044 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 7c08ab332124429889f295eb70202100 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8]), partition=
I20240502 01:58:53.154043 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 7c08ab332124429889f295eb70202100 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8]), partition=
I20240502 01:58:53.154213 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c08ab332124429889f295eb70202100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.154276 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c08ab332124429889f295eb70202100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.154275 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c08ab332124429889f295eb70202100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.156479 540 tablet_bootstrap.cc:492] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:53.156672 537 tablet_bootstrap.cc:492] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:53.157562 537 tablet_bootstrap.cc:654] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.157661 540 tablet_bootstrap.cc:654] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.158617 538 tablet_bootstrap.cc:492] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:53.158880 537 tablet_bootstrap.cc:492] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:53.159009 537 ts_tablet_manager.cc:1367] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:53.159058 540 tablet_bootstrap.cc:492] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:53.159173 540 ts_tablet_manager.cc:1367] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:53.159482 538 tablet_bootstrap.cc:654] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.159741 540 raft_consensus.cc:348] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.159848 540 raft_consensus.cc:374] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.159953 540 raft_consensus.cc:729] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.159485 537 raft_consensus.cc:348] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.160138 537 raft_consensus.cc:374] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.160113 540 consensus_queue.cc:260] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.160208 537 raft_consensus.cc:729] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.160405 540 ts_tablet_manager.cc:1398] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:53.160423 538 tablet_bootstrap.cc:492] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:53.160444 537 consensus_queue.cc:260] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.160513 538 ts_tablet_manager.cc:1367] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:53.160705 537 ts_tablet_manager.cc:1398] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:53.161022 538 raft_consensus.cc:348] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.161115 538 raft_consensus.cc:374] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.161142 538 raft_consensus.cc:729] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.161250 538 consensus_queue.cc:260] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.161548 538 ts_tablet_manager.cc:1398] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:53.190847 24763 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:53.191218 531 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:53.220405 531 consensus_queue.cc:1035] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.228257 24896 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:53.228595 531 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:53.230619 27578 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 01:58:53.230885 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35596116445411f857f52eadd963a16: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:53.231385 32105 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:53.245194 32740 consensus_queue.cc:1035] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.248726 32740 consensus_queue.cc:1035] T 6f457b933f4a41b2b5588e767d76e325 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.252857 32147 raft_consensus.cc:482] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:53.252945 32147 raft_consensus.cc:504] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.253294 32147 leader_election.cc:288] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:53.253695 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c08ab332124429889f295eb70202100" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:53.253698 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c08ab332124429889f295eb70202100" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:53.253813 24632 raft_consensus.cc:2454] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:53.253811 27176 raft_consensus.cc:2454] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:53.254132 24700 leader_election.cc:302] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:53.254379 32147 raft_consensus.cc:2781] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:53.254442 32147 raft_consensus.cc:482] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:53.254495 32147 raft_consensus.cc:3037] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.255540 32147 raft_consensus.cc:504] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.255806 32147 leader_election.cc:288] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:53.256055 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c08ab332124429889f295eb70202100" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:53.256145 27176 raft_consensus.cc:3037] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.256136 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c08ab332124429889f295eb70202100" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:53.256243 24632 raft_consensus.cc:3037] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.256938 27176 raft_consensus.cc:2454] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:53.256978 24632 raft_consensus.cc:2454] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:53.257297 24700 leader_election.cc:302] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:53.257467 32147 raft_consensus.cc:2781] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:53.257614 32147 raft_consensus.cc:686] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:53.257751 32147 consensus_queue.cc:237] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:53.259744 24483 catalog_manager.cc:5484] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:53.291162 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:53.297463 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 4f23ff531f624dacb017dd4edd7372b1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.297546 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 4f23ff531f624dacb017dd4edd7372b1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.297660 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb666b32dc84ee0b5ac33a3652688c2 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.297760 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f23ff531f624dacb017dd4edd7372b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.297806 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f23ff531f624dacb017dd4edd7372b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.297879 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb666b32dc84ee0b5ac33a3652688c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.297808 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 4f23ff531f624dacb017dd4edd7372b1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.298070 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f23ff531f624dacb017dd4edd7372b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.298637 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb666b32dc84ee0b5ac33a3652688c2 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.298880 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb666b32dc84ee0b5ac33a3652688c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.300141 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb666b32dc84ee0b5ac33a3652688c2 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.300235 537 tablet_bootstrap.cc:492] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:53.300379 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb666b32dc84ee0b5ac33a3652688c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.300873 582 tablet_bootstrap.cc:492] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:53.301144 537 tablet_bootstrap.cc:654] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.301863 582 tablet_bootstrap.cc:654] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.302688 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 71cf0698e8424399988a5cbde85eb2b4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.302768 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 71cf0698e8424399988a5cbde85eb2b4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.302901 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cf0698e8424399988a5cbde85eb2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.302940 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cf0698e8424399988a5cbde85eb2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.303292 583 tablet_bootstrap.cc:492] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:53.304077 583 tablet_bootstrap.cc:654] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.304617 538 tablet_bootstrap.cc:492] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:53.304801 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 71cf0698e8424399988a5cbde85eb2b4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=680e7d401e814f65946e6c65cd6ec986]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:53.305007 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cf0698e8424399988a5cbde85eb2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.305225 583 tablet_bootstrap.cc:492] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:53.305294 583 ts_tablet_manager.cc:1367] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:53.305444 538 tablet_bootstrap.cc:654] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.305668 583 raft_consensus.cc:348] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.305752 583 raft_consensus.cc:374] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.305778 583 raft_consensus.cc:729] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.305871 583 consensus_queue.cc:260] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.306139 583 ts_tablet_manager.cc:1398] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:53.306720 538 tablet_bootstrap.cc:492] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:53.306790 538 ts_tablet_manager.cc:1367] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:53.307212 538 raft_consensus.cc:348] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.307307 538 raft_consensus.cc:374] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.307335 538 raft_consensus.cc:729] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.307448 537 tablet_bootstrap.cc:492] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:53.307440 538 consensus_queue.cc:260] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.307523 537 ts_tablet_manager.cc:1367] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240502 01:58:53.307713 538 ts_tablet_manager.cc:1398] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:53.307870 538 tablet_bootstrap.cc:492] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:53.308034 537 raft_consensus.cc:348] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.308151 537 raft_consensus.cc:374] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.308187 537 raft_consensus.cc:729] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.308310 537 consensus_queue.cc:260] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.308605 537 ts_tablet_manager.cc:1398] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:53.308702 538 tablet_bootstrap.cc:654] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.308768 537 tablet_bootstrap.cc:492] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:53.309891 585 tablet_bootstrap.cc:492] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:53.310397 582 tablet_bootstrap.cc:492] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:53.310494 582 ts_tablet_manager.cc:1367] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:53.310925 582 raft_consensus.cc:348] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.311013 582 raft_consensus.cc:374] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.311041 582 raft_consensus.cc:729] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.311148 585 tablet_bootstrap.cc:654] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.311169 582 consensus_queue.cc:260] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.311471 582 ts_tablet_manager.cc:1398] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:53.311561 537 tablet_bootstrap.cc:654] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.311671 582 tablet_bootstrap.cc:492] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:53.312494 582 tablet_bootstrap.cc:654] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.313405 582 tablet_bootstrap.cc:492] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:53.313549 582 ts_tablet_manager.cc:1367] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:53.314030 538 tablet_bootstrap.cc:492] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:53.314116 538 ts_tablet_manager.cc:1367] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:53.314091 582 raft_consensus.cc:348] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.314185 582 raft_consensus.cc:374] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.314211 582 raft_consensus.cc:729] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.314555 538 raft_consensus.cc:348] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.314594 582 consensus_queue.cc:260] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.314666 538 raft_consensus.cc:374] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.314700 538 raft_consensus.cc:729] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.314811 538 consensus_queue.cc:260] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.315127 538 ts_tablet_manager.cc:1398] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:53.315183 585 tablet_bootstrap.cc:492] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:53.315227 582 ts_tablet_manager.cc:1398] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:53.315256 585 ts_tablet_manager.cc:1367] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:58:53.315376 582 tablet_bootstrap.cc:492] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:53.315691 585 raft_consensus.cc:348] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.315786 585 raft_consensus.cc:374] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.315812 585 raft_consensus.cc:729] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.315917 585 consensus_queue.cc:260] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.316190 585 ts_tablet_manager.cc:1398] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:53.316529 582 tablet_bootstrap.cc:654] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.317953 582 tablet_bootstrap.cc:492] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:53.318253 582 ts_tablet_manager.cc:1367] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:53.318734 582 raft_consensus.cc:348] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.318877 582 raft_consensus.cc:374] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.318935 582 raft_consensus.cc:729] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.319072 582 consensus_queue.cc:260] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.319288 537 tablet_bootstrap.cc:492] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:53.319380 537 ts_tablet_manager.cc:1367] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:58:53.319420 582 ts_tablet_manager.cc:1398] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:53.319790 537 raft_consensus.cc:348] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.319895 537 raft_consensus.cc:374] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.319936 537 raft_consensus.cc:729] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.320045 537 consensus_queue.cc:260] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.320302 537 ts_tablet_manager.cc:1398] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:58:53.322438 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:53.358625 32740 raft_consensus.cc:482] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:53.358727 32740 raft_consensus.cc:504] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.359141 32740 leader_election.cc:288] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:53.359586 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb666b32dc84ee0b5ac33a3652688c2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:53.359709 27578 raft_consensus.cc:2454] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:53.359890 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb666b32dc84ee0b5ac33a3652688c2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:53.360014 27445 raft_consensus.cc:2454] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:53.360278 24966 leader_election.cc:302] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:53.360432 32740 raft_consensus.cc:2781] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:53.360492 32740 raft_consensus.cc:482] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:53.360528 32740 raft_consensus.cc:3037] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.361531 32740 raft_consensus.cc:504] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.361820 32740 leader_election.cc:288] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:53.362085 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb666b32dc84ee0b5ac33a3652688c2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:53.362187 27578 raft_consensus.cc:3037] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.362205 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb666b32dc84ee0b5ac33a3652688c2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:53.362303 27445 raft_consensus.cc:3037] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.363065 27578 raft_consensus.cc:2454] T 5fb666b32dc84ee0b5ac33a3652688c2 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:53.363147 27445 raft_consensus.cc:2454] T 5fb666b32dc84ee0b5ac33a3652688c2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:53.363360 24966 leader_election.cc:302] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:53.363521 32740 raft_consensus.cc:2781] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:53.363598 32740 raft_consensus.cc:686] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:53.363718 32740 consensus_queue.cc:237] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:53.365463 24482 catalog_manager.cc:5484] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:53.377586 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240502 01:58:53.379302 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:53.380879 27176 raft_consensus.cc:1261] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:53.380955 24632 raft_consensus.cc:1261] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:53.381330 580 consensus_queue.cc:1035] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.381572 32147 consensus_queue.cc:1035] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.384106 24482 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 7c08ab332124429889f295eb70202100 (table rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8]). Expected version 1 got 0
I20240502 01:58:53.384409 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 7c08ab332124429889f295eb70202100 (table rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8]). Expected version 1 got 0
I20240502 01:58:53.385053 480 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:53.384925 600 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:53.385339 480 tablet.cc:1620] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:53.385438 600 tablet.cc:1620] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:53.387451 599 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:53.387686 24482 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 7c08ab332124429889f295eb70202100 (table rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8]). Expected version 1 got 0
I20240502 01:58:53.387794 599 tablet.cc:1620] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:53.389899 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8] alter complete (version 1)
W20240502 01:58:53.411636 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:53.436753 32100 raft_consensus.cc:482] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:53.436877 32100 raft_consensus.cc:504] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.437261 32100 leader_election.cc:288] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:53.437628 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f23ff531f624dacb017dd4edd7372b1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:53.437762 24763 raft_consensus.cc:2454] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:53.438055 27514 leader_election.cc:302] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:53.438074 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f23ff531f624dacb017dd4edd7372b1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:53.438184 27176 raft_consensus.cc:2454] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:53.438493 32100 raft_consensus.cc:2781] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:53.438555 32100 raft_consensus.cc:482] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:53.438596 32100 raft_consensus.cc:3037] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.439458 32100 raft_consensus.cc:504] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.439793 32100 leader_election.cc:288] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:53.440119 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f23ff531f624dacb017dd4edd7372b1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:53.440236 27176 raft_consensus.cc:3037] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.440237 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f23ff531f624dacb017dd4edd7372b1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:53.440371 24763 raft_consensus.cc:3037] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.441190 24763 raft_consensus.cc:2454] T 4f23ff531f624dacb017dd4edd7372b1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:53.441221 27176 raft_consensus.cc:2454] T 4f23ff531f624dacb017dd4edd7372b1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:53.441525 27513 leader_election.cc:302] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:53.441668 32100 raft_consensus.cc:2781] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:53.441742 32100 raft_consensus.cc:686] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:53.441864 32100 consensus_queue.cc:237] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.443706 24482 catalog_manager.cc:5484] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:53.493695 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:53.498275 599 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:53.498622 599 tablet.cc:1620] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:53.498801 600 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:53.499027 480 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:53.499125 600 tablet.cc:1620] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:53.499280 480 tablet.cc:1620] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:53.503589 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9cafe2151a67427b86b2f8afde08a3c8] alter complete (version 2)
W20240502 01:58:53.533254 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:53.583618 593 raft_consensus.cc:482] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:53.583715 593 raft_consensus.cc:504] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.584045 593 leader_election.cc:288] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:53.584329 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cf0698e8424399988a5cbde85eb2b4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:53.584342 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cf0698e8424399988a5cbde85eb2b4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:53.584441 24763 raft_consensus.cc:2454] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:53.584465 27578 raft_consensus.cc:2454] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:53.584679 27113 leader_election.cc:302] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:53.584836 593 raft_consensus.cc:2781] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:53.584888 593 raft_consensus.cc:482] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:53.584939 593 raft_consensus.cc:3037] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.585732 593 raft_consensus.cc:504] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.586035 593 leader_election.cc:288] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:53.586356 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cf0698e8424399988a5cbde85eb2b4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:53.586365 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cf0698e8424399988a5cbde85eb2b4" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:53.586462 27578 raft_consensus.cc:3037] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.586493 24763 raft_consensus.cc:3037] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.587159 27578 raft_consensus.cc:2454] T 71cf0698e8424399988a5cbde85eb2b4 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:53.587384 24763 raft_consensus.cc:2454] T 71cf0698e8424399988a5cbde85eb2b4 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:53.587402 27113 leader_election.cc:302] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:53.587553 593 raft_consensus.cc:2781] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:53.587705 593 raft_consensus.cc:686] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:53.587827 593 consensus_queue.cc:237] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:53.589879 24482 catalog_manager.cc:5484] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:53.606180 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:53.612116 600 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:53.612533 480 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:53.612640 599 tablet.cc:1722] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:53.616636 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=9cafe2151a67427b86b2f8afde08a3c8] alter complete (version 3)
I20240502 01:58:53.674906 27446 consensus_queue.cc:237] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:53.675876 27577 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:53.675946 24763 raft_consensus.cc:1261] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:53.676244 32105 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:53.676432 531 consensus_queue.cc:1035] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:53.678277 32105 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:53.678874 24763 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P 92cb785c165749eca49af5784756461c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:53.680152 27577 raft_consensus.cc:2932] T d5ec1158a7414039b6f4bc855207c938 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:53.681535 24482 catalog_manager.cc:5484] T d5ec1158a7414039b6f4bc855207c938 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 2 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:53.683064 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5ec1158a7414039b6f4bc855207c938 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:53.684173 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ec1158a7414039b6f4bc855207c938 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:53.684676 667 tablet_replica.cc:330] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:53.684837 667 raft_consensus.cc:2229] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:53.684957 667 raft_consensus.cc:2258] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:53.685606 667 ts_tablet_manager.cc:1875] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:53.687654 667 ts_tablet_manager.cc:1888] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:53.687722 667 log.cc:1196] T d5ec1158a7414039b6f4bc855207c938 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5ec1158a7414039b6f4bc855207c938
I20240502 01:58:53.688170 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet d5ec1158a7414039b6f4bc855207c938 (table rebalance_test_table_19 [id=48242a8b49174baba8323ad9a482449a]) successfully deleted
I20240502 01:58:53.719991 24483 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:53.720094 24483 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:53.721674 24483 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 7c08ab332124429889f295eb70202100
I20240502 01:58:53.722119 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c08ab332124429889f295eb70202100 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:53 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:53.722169 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c08ab332124429889f295eb70202100 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:53 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:53.722265 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c08ab332124429889f295eb70202100 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:53 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:53.722434 668 tablet_replica.cc:330] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:53.722504 669 tablet_replica.cc:330] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:53.722599 668 raft_consensus.cc:2229] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:53.722707 670 tablet_replica.cc:330] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:53.722764 669 raft_consensus.cc:2229] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:53.722879 668 raft_consensus.cc:2258] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:53.722962 670 raft_consensus.cc:2229] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:53.723119 670 raft_consensus.cc:2258] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:53.723362 669 raft_consensus.cc:2258] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:53.723469 668 ts_tablet_manager.cc:1875] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:53.723958 670 ts_tablet_manager.cc:1875] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:53.724018 669 ts_tablet_manager.cc:1875] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:53.725549 668 ts_tablet_manager.cc:1888] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:53.725613 668 log.cc:1196] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c08ab332124429889f295eb70202100
I20240502 01:58:53.725863 668 ts_tablet_manager.cc:1909] T 7c08ab332124429889f295eb70202100 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:53.726171 670 ts_tablet_manager.cc:1888] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:53.726171 669 ts_tablet_manager.cc:1888] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:53.726245 670 log.cc:1196] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/7c08ab332124429889f295eb70202100
I20240502 01:58:53.726245 669 log.cc:1196] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/7c08ab332124429889f295eb70202100
I20240502 01:58:53.726444 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 7c08ab332124429889f295eb70202100 (table rebalancer_dynamic_table_new_name [id=9cafe2151a67427b86b2f8afde08a3c8]) successfully deleted
I20240502 01:58:53.726490 669 ts_tablet_manager.cc:1909] T 7c08ab332124429889f295eb70202100 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:53.726622 670 ts_tablet_manager.cc:1909] T 7c08ab332124429889f295eb70202100 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:53.727054 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 7c08ab332124429889f295eb70202100 (table rebalancer_dynamic_table_new_name [id=9cafe2151a67427b86b2f8afde08a3c8]) successfully deleted
I20240502 01:58:53.727195 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 7c08ab332124429889f295eb70202100 (table rebalancer_dynamic_table_new_name [id=9cafe2151a67427b86b2f8afde08a3c8]) successfully deleted
I20240502 01:58:53.783206 577 consensus_queue.cc:1035] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.785965 666 consensus_queue.cc:1035] T 5fb666b32dc84ee0b5ac33a3652688c2 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.823243 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:53.828227 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 12e3c796fff743588d03c18e5042f0ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4]), partition=
I20240502 01:58:53.828510 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e3c796fff743588d03c18e5042f0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.830346 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 12e3c796fff743588d03c18e5042f0ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4]), partition=
I20240502 01:58:53.830623 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e3c796fff743588d03c18e5042f0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.830863 676 tablet_bootstrap.cc:492] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:53.831691 676 tablet_bootstrap.cc:654] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.832557 678 tablet_bootstrap.cc:492] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:53.832737 676 tablet_bootstrap.cc:492] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:53.832731 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 12e3c796fff743588d03c18e5042f0ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4]), partition=
I20240502 01:58:53.832829 676 ts_tablet_manager.cc:1367] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:53.832962 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e3c796fff743588d03c18e5042f0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.833289 676 raft_consensus.cc:348] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.833390 676 raft_consensus.cc:374] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.833420 676 raft_consensus.cc:729] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.833583 678 tablet_bootstrap.cc:654] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.833701 676 consensus_queue.cc:260] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.834709 678 tablet_bootstrap.cc:492] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:53.834744 676 ts_tablet_manager.cc:1398] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:53.834781 678 ts_tablet_manager.cc:1367] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:53.835023 680 tablet_bootstrap.cc:492] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:53.835559 678 raft_consensus.cc:348] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.835690 678 raft_consensus.cc:374] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.835739 678 raft_consensus.cc:729] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.835867 678 consensus_queue.cc:260] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.836182 678 ts_tablet_manager.cc:1398] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:58:53.837368 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:53.837519 680 tablet_bootstrap.cc:654] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:53.839038 680 tablet_bootstrap.cc:492] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:53.839123 680 ts_tablet_manager.cc:1367] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:53.839838 680 raft_consensus.cc:348] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.839942 680 raft_consensus.cc:374] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:53.839972 680 raft_consensus.cc:729] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.840118 680 consensus_queue.cc:260] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.840442 680 ts_tablet_manager.cc:1398] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:53.841228 32693 raft_consensus.cc:482] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:53.841311 32693 raft_consensus.cc:504] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.841648 32693 leader_election.cc:288] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:53.841882 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e3c796fff743588d03c18e5042f0ca" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:53.842043 27177 raft_consensus.cc:2454] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:53.842123 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e3c796fff743588d03c18e5042f0ca" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:53.842249 24763 raft_consensus.cc:2454] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:53.842391 24566 leader_election.cc:302] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:53.842669 32693 raft_consensus.cc:2781] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:53.842726 32693 raft_consensus.cc:482] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:53.842761 32693 raft_consensus.cc:3037] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.843740 32693 raft_consensus.cc:504] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.844110 32693 leader_election.cc:288] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:53.844441 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e3c796fff743588d03c18e5042f0ca" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:53.844547 27177 raft_consensus.cc:3037] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.844627 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e3c796fff743588d03c18e5042f0ca" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:53.844738 24763 raft_consensus.cc:3037] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:53.845561 27177 raft_consensus.cc:2454] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:53.845671 24763 raft_consensus.cc:2454] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:53.845991 24567 leader_election.cc:302] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:53.846272 683 raft_consensus.cc:2781] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:53.846356 683 raft_consensus.cc:686] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:53.846498 683 consensus_queue.cc:237] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:53.848316 24482 catalog_manager.cc:5484] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:53.863140 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:53.922832 24897 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:53.924226 24632 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:58:53.924477 24833 consensus_peers.cc:485] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b -> Peer bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): Couldn't send request to peer bb653087e8554c7591cf547dd969f3be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93695e2165e142f79293ac14e4f4fb84. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:53.924698 27446 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:53.924846 722 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:53.925040 394 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:53.926811 722 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:53.927234 24632 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:53.927235 27446 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:53.929905 24482 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: config changed from index 6 to 7, NON_VOTER bb653087e8554c7591cf547dd969f3be (127.17.243.8) added. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } } }
I20240502 01:58:53.955016 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:53.959713 24763 raft_consensus.cc:1261] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:53.959721 27177 raft_consensus.cc:1261] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:53.960206 683 consensus_queue.cc:1035] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.960422 32693 consensus_queue.cc:1035] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.962942 24486 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 12e3c796fff743588d03c18e5042f0ca (table rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4]). Expected version 1 got 0
I20240502 01:58:53.963019 600 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:53.963306 600 tablet.cc:1620] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:53.964036 599 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:53.964357 599 tablet.cc:1620] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:53.965080 24486 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 12e3c796fff743588d03c18e5042f0ca (table rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4]). Expected version 1 got 0
W20240502 01:58:53.966223 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:53.969310 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4] alter complete (version 1)
I20240502 01:58:53.970001 480 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:53.970331 480 tablet.cc:1620] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:53.985704 32100 consensus_queue.cc:1035] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.988458 31949 consensus_queue.cc:1035] T 4f23ff531f624dacb017dd4edd7372b1 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.995616 32084 consensus_queue.cc:1035] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:53.998867 593 consensus_queue.cc:1035] T 71cf0698e8424399988a5cbde85eb2b4 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:54.028386 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:54.070622 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:54.075853 600 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:54.076202 600 tablet.cc:1620] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:54.076290 480 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:54.076588 480 tablet.cc:1620] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:54.076761 599 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:54.077055 599 tablet.cc:1620] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:54.082464 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=49e8c0fb43e640d2940ff8dd0f20fad4] alter complete (version 2)
I20240502 01:58:54.105463 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:54.111042 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed5792d616546a8bb980f3785e5708c (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.111352 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed5792d616546a8bb980f3785e5708c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.111346 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed5792d616546a8bb980f3785e5708c (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.111621 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed5792d616546a8bb980f3785e5708c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.112059 25012 tablet_service.cc:1467] Processing CreateTablet for tablet d753987d3c9744298ee01b3bfe8c4a13 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.112336 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d753987d3c9744298ee01b3bfe8c4a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.113437 765 tablet_bootstrap.cc:492] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:54.113588 676 tablet_bootstrap.cc:492] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:54.113970 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 2a053966744742d19d808d800e9fb42b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.114190 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a053966744742d19d808d800e9fb42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.114604 765 tablet_bootstrap.cc:654] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.114609 676 tablet_bootstrap.cc:654] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.115293 766 tablet_bootstrap.cc:492] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:54.115693 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 2a053966744742d19d808d800e9fb42b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.115865 765 tablet_bootstrap.cc:492] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:54.115957 765 ts_tablet_manager.cc:1367] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:54.116305 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a053966744742d19d808d800e9fb42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.116334 27156 tablet_service.cc:1467] Processing CreateTablet for tablet d753987d3c9744298ee01b3bfe8c4a13 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.116402 765 raft_consensus.cc:348] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.116516 765 raft_consensus.cc:374] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.116554 765 raft_consensus.cc:729] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.116597 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d753987d3c9744298ee01b3bfe8c4a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.116670 765 consensus_queue.cc:260] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.116981 765 ts_tablet_manager.cc:1398] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:54.117039 27424 tablet_service.cc:1467] Processing CreateTablet for tablet d753987d3c9744298ee01b3bfe8c4a13 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.117151 765 tablet_bootstrap.cc:492] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:54.117322 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d753987d3c9744298ee01b3bfe8c4a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.117480 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed5792d616546a8bb980f3785e5708c (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.117713 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed5792d616546a8bb980f3785e5708c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.118727 680 tablet_bootstrap.cc:492] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:54.118933 766 tablet_bootstrap.cc:654] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.119724 680 tablet_bootstrap.cc:654] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.119807 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 2a053966744742d19d808d800e9fb42b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2826ad78f971427fa999989e7ef5a172]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.120044 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a053966744742d19d808d800e9fb42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.120260 676 tablet_bootstrap.cc:492] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:54.120352 676 ts_tablet_manager.cc:1367] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:54.120817 676 raft_consensus.cc:348] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.120908 680 tablet_bootstrap.cc:492] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:54.120931 676 raft_consensus.cc:374] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.120965 676 raft_consensus.cc:729] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.120983 680 ts_tablet_manager.cc:1367] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:54.121094 676 consensus_queue.cc:260] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.121400 676 ts_tablet_manager.cc:1398] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:54.121462 680 raft_consensus.cc:348] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.121556 680 raft_consensus.cc:374] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.121600 680 raft_consensus.cc:729] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.121743 680 consensus_queue.cc:260] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.122032 680 ts_tablet_manager.cc:1398] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.122200 680 tablet_bootstrap.cc:492] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:54.123284 680 tablet_bootstrap.cc:654] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.125547 680 tablet_bootstrap.cc:492] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:54.125633 680 ts_tablet_manager.cc:1367] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:54.125684 765 tablet_bootstrap.cc:654] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.126274 676 tablet_bootstrap.cc:492] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:54.126107 680 raft_consensus.cc:348] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.126349 680 raft_consensus.cc:374] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.126402 680 raft_consensus.cc:729] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.126545 771 tablet_bootstrap.cc:492] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:54.126523 680 consensus_queue.cc:260] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.126813 680 ts_tablet_manager.cc:1398] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.127197 676 tablet_bootstrap.cc:654] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.127398 771 tablet_bootstrap.cc:654] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.127727 765 tablet_bootstrap.cc:492] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:54.127810 765 ts_tablet_manager.cc:1367] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240502 01:58:54.128352 765 raft_consensus.cc:348] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.128459 765 raft_consensus.cc:374] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.128492 765 raft_consensus.cc:729] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.128633 765 consensus_queue.cc:260] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.129156 765 ts_tablet_manager.cc:1398] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.129441 765 tablet_bootstrap.cc:492] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:54.130529 765 tablet_bootstrap.cc:654] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.131675 765 tablet_bootstrap.cc:492] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:54.131758 765 ts_tablet_manager.cc:1367] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:54.132230 765 raft_consensus.cc:348] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.132478 765 raft_consensus.cc:374] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.132527 765 raft_consensus.cc:729] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.132665 765 consensus_queue.cc:260] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.133173 765 ts_tablet_manager.cc:1398] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.134425 766 tablet_bootstrap.cc:492] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:54.134533 766 ts_tablet_manager.cc:1367] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 01:58:54.134989 766 raft_consensus.cc:348] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.135102 766 raft_consensus.cc:374] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.135161 766 raft_consensus.cc:729] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.135294 766 consensus_queue.cc:260] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.135598 766 ts_tablet_manager.cc:1398] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.136328 676 tablet_bootstrap.cc:492] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:54.136430 676 ts_tablet_manager.cc:1367] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:54.136505 771 tablet_bootstrap.cc:492] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:54.136612 771 ts_tablet_manager.cc:1367] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:54.136849 676 raft_consensus.cc:348] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.136945 676 raft_consensus.cc:374] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.136973 676 raft_consensus.cc:729] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.137033 771 raft_consensus.cc:348] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.137084 676 consensus_queue.cc:260] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.137137 771 raft_consensus.cc:374] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.137166 771 raft_consensus.cc:729] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.137272 771 consensus_queue.cc:260] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.137343 676 ts_tablet_manager.cc:1398] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.137559 771 ts_tablet_manager.cc:1398] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.183224 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:54.188356 600 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:54.189301 480 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:54.189332 599 tablet.cc:1722] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:54.194291 758 raft_consensus.cc:482] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.194407 758 raft_consensus.cc:504] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.194792 758 leader_election.cc:288] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:54.195174 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=49e8c0fb43e640d2940ff8dd0f20fad4] alter complete (version 3)
I20240502 01:58:54.195255 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a053966744742d19d808d800e9fb42b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:54.195343 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a053966744742d19d808d800e9fb42b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:54.195389 27577 raft_consensus.cc:2454] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:54.195456 27446 raft_consensus.cc:2454] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:54.195992 24567 leader_election.cc:302] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:58:54.196341 758 raft_consensus.cc:2781] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.196398 758 raft_consensus.cc:482] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.196429 758 raft_consensus.cc:3037] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.197351 758 raft_consensus.cc:504] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.197650 758 leader_election.cc:288] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:54.198349 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a053966744742d19d808d800e9fb42b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:54.198467 27446 raft_consensus.cc:3037] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.199440 27446 raft_consensus.cc:2454] T 2a053966744742d19d808d800e9fb42b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:54.199697 24567 leader_election.cc:302] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:58:54.199975 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a053966744742d19d808d800e9fb42b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:54.200016 758 raft_consensus.cc:2781] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:54.200083 27577 raft_consensus.cc:3037] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.200101 758 raft_consensus.cc:686] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:54.200227 758 consensus_queue.cc:237] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.201045 27577 raft_consensus.cc:2454] T 2a053966744742d19d808d800e9fb42b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:54.202323 24479 catalog_manager.cc:5484] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:54.232895 531 raft_consensus.cc:482] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.233014 531 raft_consensus.cc:504] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.233459 531 leader_election.cc:288] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:54.233834 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753987d3c9744298ee01b3bfe8c4a13" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:54.233860 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753987d3c9744298ee01b3bfe8c4a13" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:54.234032 27177 raft_consensus.cc:2454] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:54.234030 25031 raft_consensus.cc:2454] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:54.234284 27379 leader_election.cc:302] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.234549 531 raft_consensus.cc:2781] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.234601 531 raft_consensus.cc:482] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.234628 531 raft_consensus.cc:3037] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.235824 531 raft_consensus.cc:504] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.236274 531 leader_election.cc:288] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:54.236624 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753987d3c9744298ee01b3bfe8c4a13" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:54.236657 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d753987d3c9744298ee01b3bfe8c4a13" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:54.236745 27177 raft_consensus.cc:3037] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.236745 25031 raft_consensus.cc:3037] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.237728 25031 raft_consensus.cc:2454] T d753987d3c9744298ee01b3bfe8c4a13 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:54.237728 27177 raft_consensus.cc:2454] T d753987d3c9744298ee01b3bfe8c4a13 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:54.238014 27379 leader_election.cc:302] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.238185 531 raft_consensus.cc:2781] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:54.238266 531 raft_consensus.cc:686] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:54.238413 531 consensus_queue.cc:237] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.240329 24479 catalog_manager.cc:5484] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:54.254757 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:54.281912 763 raft_consensus.cc:482] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.282063 763 raft_consensus.cc:504] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.282521 763 leader_election.cc:288] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:54.282892 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed5792d616546a8bb980f3785e5708c" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:54.282927 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed5792d616546a8bb980f3785e5708c" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:54.283035 24632 raft_consensus.cc:2454] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:54.283069 27446 raft_consensus.cc:2454] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:54.283356 27113 leader_election.cc:302] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.283519 763 raft_consensus.cc:2781] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.283564 763 raft_consensus.cc:482] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.283600 763 raft_consensus.cc:3037] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:58:54.283851 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.284715 763 raft_consensus.cc:504] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.285048 763 leader_election.cc:288] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:54.285362 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed5792d616546a8bb980f3785e5708c" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:54.285455 27446 raft_consensus.cc:3037] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.285853 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed5792d616546a8bb980f3785e5708c" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:54.285979 24632 raft_consensus.cc:3037] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.286337 27446 raft_consensus.cc:2454] T 1ed5792d616546a8bb980f3785e5708c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:54.286609 27113 leader_election.cc:302] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.286756 763 raft_consensus.cc:2781] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:54.286895 24632 raft_consensus.cc:2454] T 1ed5792d616546a8bb980f3785e5708c P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:54.286918 763 raft_consensus.cc:686] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:54.287051 763 consensus_queue.cc:237] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.289171 24479 catalog_manager.cc:5484] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:54.299702 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:54.299818 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:54.302256 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 12e3c796fff743588d03c18e5042f0ca
I20240502 01:58:54.302927 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e3c796fff743588d03c18e5042f0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:54 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:54.303184 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e3c796fff743588d03c18e5042f0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:54 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:54.303432 811 tablet_replica.cc:330] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:54.303519 812 tablet_replica.cc:330] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:54.303571 811 raft_consensus.cc:2229] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:54.303651 812 raft_consensus.cc:2229] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:54.303686 811 raft_consensus.cc:2258] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:54.303664 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e3c796fff743588d03c18e5042f0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:54 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:54.303750 812 raft_consensus.cc:2258] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:54.304097 813 tablet_replica.cc:330] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:54.304482 813 raft_consensus.cc:2229] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:54.304651 812 ts_tablet_manager.cc:1875] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:54.304702 813 raft_consensus.cc:2258] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:54.304878 811 ts_tablet_manager.cc:1875] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:54.305317 813 ts_tablet_manager.cc:1875] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:54.307148 812 ts_tablet_manager.cc:1888] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:54.307227 812 log.cc:1196] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/12e3c796fff743588d03c18e5042f0ca
I20240502 01:58:54.307271 813 ts_tablet_manager.cc:1888] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:54.307276 811 ts_tablet_manager.cc:1888] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:54.307340 811 log.cc:1196] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/12e3c796fff743588d03c18e5042f0ca
I20240502 01:58:54.307374 813 log.cc:1196] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/12e3c796fff743588d03c18e5042f0ca
I20240502 01:58:54.307545 812 ts_tablet_manager.cc:1909] T 12e3c796fff743588d03c18e5042f0ca P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:54.307691 813 ts_tablet_manager.cc:1909] T 12e3c796fff743588d03c18e5042f0ca P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:54.307698 811 ts_tablet_manager.cc:1909] T 12e3c796fff743588d03c18e5042f0ca P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:54.308226 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 12e3c796fff743588d03c18e5042f0ca (table rebalancer_dynamic_table_new_name [id=49e8c0fb43e640d2940ff8dd0f20fad4]) successfully deleted
I20240502 01:58:54.308367 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 12e3c796fff743588d03c18e5042f0ca (table rebalancer_dynamic_table_new_name [id=49e8c0fb43e640d2940ff8dd0f20fad4]) successfully deleted
I20240502 01:58:54.308387 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 12e3c796fff743588d03c18e5042f0ca (table rebalancer_dynamic_table_new_name [id=49e8c0fb43e640d2940ff8dd0f20fad4]) successfully deleted
W20240502 01:58:54.312366 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:54.319906 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:54.327021 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet 252502ad2e124996b345888706e674d5: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:54.404206 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:54.409026 24743 tablet_service.cc:1467] Processing CreateTablet for tablet ba7941497ec04f6fa79529bf11c98d7f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]), partition=
I20240502 01:58:54.409205 27157 tablet_service.cc:1467] Processing CreateTablet for tablet ba7941497ec04f6fa79529bf11c98d7f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]), partition=
I20240502 01:58:54.409212 24877 tablet_service.cc:1467] Processing CreateTablet for tablet ba7941497ec04f6fa79529bf11c98d7f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]), partition=
I20240502 01:58:54.409297 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7941497ec04f6fa79529bf11c98d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.409426 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7941497ec04f6fa79529bf11c98d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.409448 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7941497ec04f6fa79529bf11c98d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.411316 680 tablet_bootstrap.cc:492] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:54.411854 816 tablet_bootstrap.cc:492] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:54.412237 680 tablet_bootstrap.cc:654] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.412741 816 tablet_bootstrap.cc:654] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.413434 817 tablet_bootstrap.cc:492] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
W20240502 01:58:54.413443 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.413524 680 tablet_bootstrap.cc:492] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:54.413614 680 ts_tablet_manager.cc:1367] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:54.414162 680 raft_consensus.cc:348] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.414273 680 raft_consensus.cc:374] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.414305 680 raft_consensus.cc:729] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.414453 817 tablet_bootstrap.cc:654] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.414422 680 consensus_queue.cc:260] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.414661 680 ts_tablet_manager.cc:1398] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.415688 816 tablet_bootstrap.cc:492] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:54.415789 816 ts_tablet_manager.cc:1367] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:54.416213 816 raft_consensus.cc:348] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.416321 816 raft_consensus.cc:374] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.416347 816 raft_consensus.cc:729] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.416496 816 consensus_queue.cc:260] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.416779 816 ts_tablet_manager.cc:1398] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.417182 817 tablet_bootstrap.cc:492] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:54.417305 817 ts_tablet_manager.cc:1367] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:54.417737 817 raft_consensus.cc:348] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.417887 817 raft_consensus.cc:374] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.418035 817 raft_consensus.cc:729] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.418529 817 consensus_queue.cc:260] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.418808 817 ts_tablet_manager.cc:1398] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240502 01:58:54.420961 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:54.502664 722 raft_consensus.cc:482] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.502781 722 raft_consensus.cc:504] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.503129 722 leader_election.cc:288] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:54.503474 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7941497ec04f6fa79529bf11c98d7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:54.503628 24763 raft_consensus.cc:2454] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:54.503599 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7941497ec04f6fa79529bf11c98d7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:54.503736 27177 raft_consensus.cc:2454] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:54.503976 24833 leader_election.cc:302] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:54.504143 722 raft_consensus.cc:2781] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.504210 722 raft_consensus.cc:482] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.504258 722 raft_consensus.cc:3037] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.505102 722 raft_consensus.cc:504] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.505401 722 leader_election.cc:288] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:54.505657 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7941497ec04f6fa79529bf11c98d7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:54.505694 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7941497ec04f6fa79529bf11c98d7f" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:54.505765 24763 raft_consensus.cc:3037] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.505775 27177 raft_consensus.cc:3037] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.506510 27177 raft_consensus.cc:2454] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:54.506662 24763 raft_consensus.cc:2454] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:54.506760 24832 leader_election.cc:302] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:54.506917 722 raft_consensus.cc:2781] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:54.507072 722 raft_consensus.cc:686] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:54.507207 722 consensus_queue.cc:237] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.509055 24486 catalog_manager.cc:5484] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:54.535640 855 ts_tablet_manager.cc:902] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: Initiating tablet copy from peer f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:54.536188 855 tablet_copy_client.cc:308] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: tablet copy: Beginning tablet copy session from remote peer at address 127.17.243.3:45445
I20240502 01:58:54.536617 24918 tablet_copy_service.cc:140] P f46bcd2bc124471981349b1e2726253b: Received BeginTabletCopySession request for tablet 93695e2165e142f79293ac14e4f4fb84 from peer bb653087e8554c7591cf547dd969f3be ({username='slave'} at 127.17.243.8:41115)
I20240502 01:58:54.536731 24918 tablet_copy_service.cc:161] P f46bcd2bc124471981349b1e2726253b: Beginning new tablet copy session on tablet 93695e2165e142f79293ac14e4f4fb84 from peer bb653087e8554c7591cf547dd969f3be at {username='slave'} at 127.17.243.8:41115: session id = bb653087e8554c7591cf547dd969f3be-93695e2165e142f79293ac14e4f4fb84
I20240502 01:58:54.537544 24918 tablet_copy_source_session.cc:205] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:54.538389 855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93695e2165e142f79293ac14e4f4fb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.540176 855 tablet_copy_client.cc:783] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:54.540321 855 tablet_copy_client.cc:647] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:54.541371 855 tablet_copy_client.cc:523] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:54.542438 855 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
W20240502 01:58:54.559314 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:54.575820 855 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:54.576157 855 tablet_bootstrap.cc:492] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: Bootstrap complete.
I20240502 01:58:54.576330 855 ts_tablet_manager.cc:1367] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.000s
I20240502 01:58:54.576753 855 raft_consensus.cc:348] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:54.576882 855 raft_consensus.cc:729] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: LEARNER
I20240502 01:58:54.576977 855 consensus_queue.cc:260] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: NON_VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: true } }
I20240502 01:58:54.577248 855 ts_tablet_manager.cc:1398] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:54.577762 24918 tablet_copy_service.cc:342] P f46bcd2bc124471981349b1e2726253b: Request end of tablet copy session bb653087e8554c7591cf547dd969f3be-93695e2165e142f79293ac14e4f4fb84 received from {username='slave'} at 127.17.243.8:41115
I20240502 01:58:54.577847 24918 tablet_copy_service.cc:433] P f46bcd2bc124471981349b1e2726253b: ending tablet copy session bb653087e8554c7591cf547dd969f3be-93695e2165e142f79293ac14e4f4fb84 on tablet 93695e2165e142f79293ac14e4f4fb84 with peer bb653087e8554c7591cf547dd969f3be
I20240502 01:58:54.620263 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:54.623919 683 consensus_queue.cc:1035] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.626216 27177 raft_consensus.cc:1261] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:54.626309 24763 raft_consensus.cc:1261] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:54.627193 852 consensus_queue.cc:1035] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.628513 852 consensus_queue.cc:1035] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.629805 24483 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]). Expected version 1 got 0
W20240502 01:58:54.630501 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.632691 758 consensus_queue.cc:1035] T 2a053966744742d19d808d800e9fb42b P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.633046 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]). Expected version 1 got 0
I20240502 01:58:54.635874 599 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:54.636209 599 tablet.cc:1620] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:54.636966 867 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:54.638054 24486 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]). Expected version 1 got 0
I20240502 01:58:54.638319 867 tablet.cc:1620] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:54.640254 531 consensus_queue.cc:1035] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.641547 480 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:54.641847 480 tablet.cc:1620] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:54.646440 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52]). Expected version 1 got 0
I20240502 01:58:54.646688 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52] alter complete (version 1)
I20240502 01:58:54.647615 32105 consensus_queue.cc:1035] T d753987d3c9744298ee01b3bfe8c4a13 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:54.666589 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:54.673331 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.738744 851 consensus_queue.cc:1035] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.746601 763 consensus_queue.cc:1035] T 1ed5792d616546a8bb980f3785e5708c P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:54.749513 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:54.755208 867 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:54.755548 867 tablet.cc:1620] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:54.756222 599 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:54.756538 599 tablet.cc:1620] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:54.758301 480 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:54.758602 480 tablet.cc:1620] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:54.763502 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=08534a7bce8a4f6999cd656bae435b52] alter complete (version 2)
W20240502 01:58:54.782611 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:54.822770 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:54.828784 27157 tablet_service.cc:1467] Processing CreateTablet for tablet d60c2eb8dcd7429981c853f19bfaebb2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.828985 27424 tablet_service.cc:1467] Processing CreateTablet for tablet d60c2eb8dcd7429981c853f19bfaebb2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.829111 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c2eb8dcd7429981c853f19bfaebb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.829226 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c2eb8dcd7429981c853f19bfaebb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.829582 24877 tablet_service.cc:1467] Processing CreateTablet for tablet b54e83b4026f49408d3f45da92da84e2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.829806 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54e83b4026f49408d3f45da92da84e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.830207 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 02bbdefde02641d3b0c0a7950765d5a1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.830437 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bbdefde02641d3b0c0a7950765d5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.831370 680 tablet_bootstrap.cc:492] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:54.832036 817 tablet_bootstrap.cc:492] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:54.832234 680 tablet_bootstrap.cc:654] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.832636 24610 tablet_service.cc:1467] Processing CreateTablet for tablet d60c2eb8dcd7429981c853f19bfaebb2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.832883 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c2eb8dcd7429981c853f19bfaebb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.832924 906 tablet_bootstrap.cc:492] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:54.832984 817 tablet_bootstrap.cc:654] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.833372 680 tablet_bootstrap.cc:492] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:54.833461 680 ts_tablet_manager.cc:1367] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:54.833907 680 raft_consensus.cc:348] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.834074 680 raft_consensus.cc:374] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.834110 680 raft_consensus.cc:729] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.834228 680 consensus_queue.cc:260] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.834507 680 ts_tablet_manager.cc:1398] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:54.835247 817 tablet_bootstrap.cc:492] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:54.835327 817 ts_tablet_manager.cc:1367] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:58:54.835752 906 tablet_bootstrap.cc:654] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.835737 817 raft_consensus.cc:348] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.835845 817 raft_consensus.cc:374] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.835884 817 raft_consensus.cc:729] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.836019 817 consensus_queue.cc:260] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.836292 817 ts_tablet_manager.cc:1398] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:54.836326 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 02bbdefde02641d3b0c0a7950765d5a1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.836577 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bbdefde02641d3b0c0a7950765d5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.836712 909 tablet_bootstrap.cc:492] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:54.837162 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 02bbdefde02641d3b0c0a7950765d5a1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.837397 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bbdefde02641d3b0c0a7950765d5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.837581 909 tablet_bootstrap.cc:654] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:54.837915 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:54.838248 24610 tablet_service.cc:1467] Processing CreateTablet for tablet b54e83b4026f49408d3f45da92da84e2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.838459 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54e83b4026f49408d3f45da92da84e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:58:54.839390 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.839788 680 tablet_bootstrap.cc:492] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:54.840610 680 tablet_bootstrap.cc:654] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.840931 25012 tablet_service.cc:1467] Processing CreateTablet for tablet b54e83b4026f49408d3f45da92da84e2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a25610aacb83415686d1b6e8e9cbf0d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:54.841192 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54e83b4026f49408d3f45da92da84e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:54.843422 914 tablet_bootstrap.cc:492] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:54.844529 906 tablet_bootstrap.cc:492] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:54.844621 906 ts_tablet_manager.cc:1367] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:58:54.844986 906 raft_consensus.cc:348] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.845153 906 raft_consensus.cc:374] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.845187 914 tablet_bootstrap.cc:654] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.845206 906 raft_consensus.cc:729] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.845316 906 consensus_queue.cc:260] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.845698 906 ts_tablet_manager.cc:1398] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.845921 906 tablet_bootstrap.cc:492] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:54.846900 906 tablet_bootstrap.cc:654] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.848316 680 tablet_bootstrap.cc:492] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:54.848431 680 ts_tablet_manager.cc:1367] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 01:58:54.848721 906 tablet_bootstrap.cc:492] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:54.848780 906 ts_tablet_manager.cc:1367] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:54.848836 909 tablet_bootstrap.cc:492] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:54.848930 909 ts_tablet_manager.cc:1367] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240502 01:58:54.848920 680 raft_consensus.cc:348] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.849021 680 raft_consensus.cc:374] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.849056 680 raft_consensus.cc:729] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.849107 906 raft_consensus.cc:348] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.849187 906 raft_consensus.cc:374] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.849217 906 raft_consensus.cc:729] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.849182 680 consensus_queue.cc:260] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.849320 906 consensus_queue.cc:260] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.849414 909 raft_consensus.cc:348] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.849478 680 ts_tablet_manager.cc:1398] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.849514 909 raft_consensus.cc:374] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.849556 909 raft_consensus.cc:729] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.849560 906 ts_tablet_manager.cc:1398] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:54.849685 909 consensus_queue.cc:260] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.850021 909 ts_tablet_manager.cc:1398] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:54.850188 909 tablet_bootstrap.cc:492] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:54.851073 909 tablet_bootstrap.cc:654] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.852443 914 tablet_bootstrap.cc:492] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:54.852813 914 ts_tablet_manager.cc:1367] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:54.853428 909 tablet_bootstrap.cc:492] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:54.853338 914 raft_consensus.cc:348] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.853536 914 raft_consensus.cc:374] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.853592 914 raft_consensus.cc:729] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.853739 914 consensus_queue.cc:260] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.853790 909 ts_tablet_manager.cc:1367] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:54.854182 909 raft_consensus.cc:348] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.854336 909 raft_consensus.cc:374] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.854410 909 raft_consensus.cc:729] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.854566 909 consensus_queue.cc:260] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.854924 914 ts_tablet_manager.cc:1398] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:54.855024 909 ts_tablet_manager.cc:1398] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:54.855235 909 tablet_bootstrap.cc:492] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:54.856195 909 tablet_bootstrap.cc:654] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:54.857473 909 tablet_bootstrap.cc:492] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:54.857559 909 ts_tablet_manager.cc:1367] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:54.857985 909 raft_consensus.cc:348] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.858072 909 raft_consensus.cc:374] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:54.858101 909 raft_consensus.cc:729] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.858222 909 consensus_queue.cc:260] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.858457 909 ts_tablet_manager.cc:1398] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.866137 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:54.871580 867 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:54.873528 599 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:54.874503 480 tablet.cc:1722] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 01:58:54.876875 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.877640 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=08534a7bce8a4f6999cd656bae435b52] alter complete (version 3)
I20240502 01:58:54.937150 32105 raft_consensus.cc:482] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.937268 32105 raft_consensus.cc:504] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.937646 32105 leader_election.cc:288] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:54.937917 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbdefde02641d3b0c0a7950765d5a1" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:54.938074 27176 raft_consensus.cc:2454] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:54.938366 27379 leader_election.cc:302] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.938539 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbdefde02641d3b0c0a7950765d5a1" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:54.938673 24630 raft_consensus.cc:2454] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:54.939606 32105 raft_consensus.cc:2781] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.939663 32105 raft_consensus.cc:482] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.939698 32105 raft_consensus.cc:3037] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.940649 32105 raft_consensus.cc:504] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.941006 32105 leader_election.cc:288] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:54.941256 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbdefde02641d3b0c0a7950765d5a1" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:54.941303 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbdefde02641d3b0c0a7950765d5a1" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:54.941381 27176 raft_consensus.cc:3037] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.941386 24630 raft_consensus.cc:3037] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.942322 24630 raft_consensus.cc:2454] T 02bbdefde02641d3b0c0a7950765d5a1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:54.942322 27176 raft_consensus.cc:2454] T 02bbdefde02641d3b0c0a7950765d5a1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:54.942566 27379 leader_election.cc:302] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.942723 32105 raft_consensus.cc:2781] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:54.942829 32105 raft_consensus.cc:686] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:54.942932 32105 consensus_queue.cc:237] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.944865 24486 catalog_manager.cc:5484] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:54.947414 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 01:58:54.952440 868 raft_consensus.cc:482] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.952527 868 raft_consensus.cc:504] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.952829 868 leader_election.cc:288] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:54.953128 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54e83b4026f49408d3f45da92da84e2" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:54.953171 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54e83b4026f49408d3f45da92da84e2" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:54.953227 24630 raft_consensus.cc:2454] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:54.953301 25031 raft_consensus.cc:2454] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:54.953477 24832 leader_election.cc:302] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:54.953629 868 raft_consensus.cc:2781] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.953692 868 raft_consensus.cc:482] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.953728 868 raft_consensus.cc:3037] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.954640 868 raft_consensus.cc:504] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.954924 868 leader_election.cc:288] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:54.955171 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54e83b4026f49408d3f45da92da84e2" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:54.955209 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54e83b4026f49408d3f45da92da84e2" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:54.955276 24630 raft_consensus.cc:3037] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.955325 25031 raft_consensus.cc:3037] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.955911 763 raft_consensus.cc:482] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:54.955983 763 raft_consensus.cc:504] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.956173 24630 raft_consensus.cc:2454] T b54e83b4026f49408d3f45da92da84e2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:54.956218 25031 raft_consensus.cc:2454] T b54e83b4026f49408d3f45da92da84e2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:54.956270 763 leader_election.cc:288] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:54.956432 24832 leader_election.cc:302] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:54.956562 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c2eb8dcd7429981c853f19bfaebb2" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:54.956595 722 raft_consensus.cc:2781] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:54.956656 24630 raft_consensus.cc:2454] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:54.956662 722 raft_consensus.cc:686] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:54.956777 722 consensus_queue.cc:237] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:54.957006 27112 leader_election.cc:302] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.957158 763 raft_consensus.cc:2781] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:54.957212 763 raft_consensus.cc:482] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:54.957183 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c2eb8dcd7429981c853f19bfaebb2" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:54.957247 763 raft_consensus.cc:3037] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.957281 27445 raft_consensus.cc:2454] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:54.958184 763 raft_consensus.cc:504] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.958495 763 leader_election.cc:288] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:54.959060 24486 catalog_manager.cc:5484] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:54.959843 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c2eb8dcd7429981c853f19bfaebb2" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:54.959946 24630 raft_consensus.cc:3037] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.959931 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c2eb8dcd7429981c853f19bfaebb2" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:54.960052 27445 raft_consensus.cc:3037] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:54.961773 27445 raft_consensus.cc:2454] T d60c2eb8dcd7429981c853f19bfaebb2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:54.962019 24630 raft_consensus.cc:2454] T d60c2eb8dcd7429981c853f19bfaebb2 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:54.962715 27113 leader_election.cc:302] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:54.963196 763 raft_consensus.cc:2781] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:58:54.964007 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240502 01:58:54.965963 763 raft_consensus.cc:686] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:54.966111 763 consensus_queue.cc:237] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:54.968220 24486 catalog_manager.cc:5484] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:54.979140 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:58:54.979761 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:54.986079 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:54.986186 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:54.987763 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet ba7941497ec04f6fa79529bf11c98d7f
I20240502 01:58:54.988255 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7941497ec04f6fa79529bf11c98d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:54 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:54.988269 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7941497ec04f6fa79529bf11c98d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:54 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:54.988579 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7941497ec04f6fa79529bf11c98d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:54 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:54.988910 954 tablet_replica.cc:330] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:54.989076 952 tablet_replica.cc:330] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:54.989107 954 raft_consensus.cc:2229] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:54.989187 953 tablet_replica.cc:330] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:54.989255 952 raft_consensus.cc:2229] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:54.989341 954 raft_consensus.cc:2258] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:54.989336 953 raft_consensus.cc:2229] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:54.989380 952 raft_consensus.cc:2258] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:54.989471 953 raft_consensus.cc:2258] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:54.990001 952 ts_tablet_manager.cc:1875] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:54.990078 954 ts_tablet_manager.cc:1875] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:54.990078 953 ts_tablet_manager.cc:1875] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:54.991909 952 ts_tablet_manager.cc:1888] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:54.991981 952 log.cc:1196] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba7941497ec04f6fa79529bf11c98d7f
I20240502 01:58:54.992245 954 ts_tablet_manager.cc:1888] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:54.992254 953 ts_tablet_manager.cc:1888] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:54.992285 952 ts_tablet_manager.cc:1909] T ba7941497ec04f6fa79529bf11c98d7f P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:54.992324 953 log.cc:1196] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/ba7941497ec04f6fa79529bf11c98d7f
I20240502 01:58:54.992324 954 log.cc:1196] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba7941497ec04f6fa79529bf11c98d7f
I20240502 01:58:54.992602 953 ts_tablet_manager.cc:1909] T ba7941497ec04f6fa79529bf11c98d7f P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:54.992698 954 ts_tablet_manager.cc:1909] T ba7941497ec04f6fa79529bf11c98d7f P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:54.992875 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table_new_name [id=08534a7bce8a4f6999cd656bae435b52]) successfully deleted
I20240502 01:58:54.993330 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table_new_name [id=08534a7bce8a4f6999cd656bae435b52]) successfully deleted
W20240502 01:58:54.993486 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 01:58:54.993851 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet ba7941497ec04f6fa79529bf11c98d7f (table rebalancer_dynamic_table_new_name [id=08534a7bce8a4f6999cd656bae435b52]) successfully deleted
W20240502 01:58:54.994153 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.019485 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:58:55.020272 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.029388 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 01:58:55.031113 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:58:55.031658 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.047793 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.055847 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.064736 27577 raft_consensus.cc:1203] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 01:58:55.067948 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.085601 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.089812 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:55.094710 27558 tablet_service.cc:1467] Processing CreateTablet for tablet c1ab34baf78f47f290cbbc464197e013 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]), partition=
I20240502 01:58:55.094836 27157 tablet_service.cc:1467] Processing CreateTablet for tablet c1ab34baf78f47f290cbbc464197e013 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]), partition=
I20240502 01:58:55.094988 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ab34baf78f47f290cbbc464197e013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.095044 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ab34baf78f47f290cbbc464197e013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.097085 680 tablet_bootstrap.cc:492] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:55.097185 955 tablet_bootstrap.cc:492] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:55.097990 680 tablet_bootstrap.cc:654] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.097991 955 tablet_bootstrap.cc:654] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.098731 24877 tablet_service.cc:1467] Processing CreateTablet for tablet c1ab34baf78f47f290cbbc464197e013 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]), partition=
I20240502 01:58:55.098964 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ab34baf78f47f290cbbc464197e013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.099077 680 tablet_bootstrap.cc:492] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:55.099164 680 ts_tablet_manager.cc:1367] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:55.099581 680 raft_consensus.cc:348] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.099686 680 raft_consensus.cc:374] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.099725 680 raft_consensus.cc:729] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.099875 680 consensus_queue.cc:260] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.100399 680 ts_tablet_manager.cc:1398] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:55.100937 955 tablet_bootstrap.cc:492] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:55.101033 955 ts_tablet_manager.cc:1367] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:55.101114 817 tablet_bootstrap.cc:492] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:55.101466 955 raft_consensus.cc:348] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.101598 955 raft_consensus.cc:374] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.101637 955 raft_consensus.cc:729] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.101750 955 consensus_queue.cc:260] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.102052 817 tablet_bootstrap.cc:654] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.102058 955 ts_tablet_manager.cc:1398] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:55.103019 817 tablet_bootstrap.cc:492] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:55.103086 817 ts_tablet_manager.cc:1367] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:55.103432 817 raft_consensus.cc:348] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.103520 817 raft_consensus.cc:374] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.103540 817 raft_consensus.cc:729] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.103652 817 consensus_queue.cc:260] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.103896 817 ts_tablet_manager.cc:1398] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:58:55.162089 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.192147 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.221918 760 raft_consensus.cc:482] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:55.222066 760 raft_consensus.cc:504] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.222493 760 leader_election.cc:288] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:55.222769 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ab34baf78f47f290cbbc464197e013" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:55.222790 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ab34baf78f47f290cbbc464197e013" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:55.222903 27177 raft_consensus.cc:2454] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:55.222918 24897 raft_consensus.cc:2454] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:55.223150 27513 leader_election.cc:302] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:55.223330 760 raft_consensus.cc:2781] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:55.223385 760 raft_consensus.cc:482] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:55.223420 760 raft_consensus.cc:3037] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.224262 760 raft_consensus.cc:504] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.224552 760 leader_election.cc:288] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:55.224835 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ab34baf78f47f290cbbc464197e013" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:55.224893 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ab34baf78f47f290cbbc464197e013" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:55.224961 24897 raft_consensus.cc:3037] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.224982 27177 raft_consensus.cc:3037] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.225829 24897 raft_consensus.cc:2454] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:55.225829 27177 raft_consensus.cc:2454] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:55.226105 27513 leader_election.cc:302] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:55.226263 760 raft_consensus.cc:2781] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:55.226348 760 raft_consensus.cc:686] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:55.226645 760 consensus_queue.cc:237] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:55.228549 24479 catalog_manager.cc:5484] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:55.326809 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 01:58:55.339316 991 consensus_queue.cc:1035] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:55.339598 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.343003 991 consensus_queue.cc:1035] T 02bbdefde02641d3b0c0a7950765d5a1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:55.347244 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.353627 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:55.357278 27177 raft_consensus.cc:1261] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:55.357331 24897 raft_consensus.cc:1261] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:55.357659 760 consensus_queue.cc:1035] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:55.357833 32100 consensus_queue.cc:1035] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:55.360594 24486 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]). Expected version 1 got 0
I20240502 01:58:55.361369 867 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:55.361685 867 tablet.cc:1620] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:55.362560 480 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:55.362756 24479 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]). Expected version 1 got 0
I20240502 01:58:55.362855 480 tablet.cc:1620] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:55.363884 24486 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]). Expected version 1 got 0
I20240502 01:58:55.365914 1001 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:55.366387 1001 tablet.cc:1620] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:55.367120 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c] alter complete (version 1)
I20240502 01:58:55.367667 24486 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c]). Expected version 1 got 0
W20240502 01:58:55.368237 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:55.404479 974 consensus_queue.cc:1035] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:55.407835 974 consensus_queue.cc:1035] T d60c2eb8dcd7429981c853f19bfaebb2 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:55.441789 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.470139 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:55.475270 1001 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:55.475615 1001 tablet.cc:1620] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:55.475736 867 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:55.476054 867 tablet.cc:1620] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:55.476395 480 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:55.476680 480 tablet.cc:1620] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:55.482025 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=a11c1219ccf249a0bc158ed83235440c] alter complete (version 2)
W20240502 01:58:55.496835 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.498450 24483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:55.503630 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5760b6d6b450d9bc31503aec7e6e4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.503916 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a5760b6d6b450d9bc31503aec7e6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.504019 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 30939882dceb4f408b5999843644105c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.504288 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30939882dceb4f408b5999843644105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.504364 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 30939882dceb4f408b5999843644105c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.504516 24744 tablet_service.cc:1467] Processing CreateTablet for tablet deeb5586ff92483a9b85dc8631adc63c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.504591 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30939882dceb4f408b5999843644105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.504756 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deeb5586ff92483a9b85dc8631adc63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.506228 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5760b6d6b450d9bc31503aec7e6e4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.506471 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a5760b6d6b450d9bc31503aec7e6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.506928 1036 tablet_bootstrap.cc:492] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:55.507004 680 tablet_bootstrap.cc:492] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:55.507851 1036 tablet_bootstrap.cc:654] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.507876 680 tablet_bootstrap.cc:654] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.508483 817 tablet_bootstrap.cc:492] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:55.508893 680 tablet_bootstrap.cc:492] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:55.508977 680 ts_tablet_manager.cc:1367] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:55.508903 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 30939882dceb4f408b5999843644105c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.509171 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30939882dceb4f408b5999843644105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.509126 25012 tablet_service.cc:1467] Processing CreateTablet for tablet deeb5586ff92483a9b85dc8631adc63c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.509331 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deeb5586ff92483a9b85dc8631adc63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.509377 817 tablet_bootstrap.cc:654] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.509389 680 raft_consensus.cc:348] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.509486 680 raft_consensus.cc:374] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.509516 680 raft_consensus.cc:729] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.509634 680 consensus_queue.cc:260] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.509874 680 ts_tablet_manager.cc:1398] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:55.508903 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5760b6d6b450d9bc31503aec7e6e4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.510862 24877 tablet_service.cc:1467] Processing CreateTablet for tablet deeb5586ff92483a9b85dc8631adc63c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=fd37025413ec443884b06cf7e7d57569]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:55.510957 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a5760b6d6b450d9bc31503aec7e6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.511060 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deeb5586ff92483a9b85dc8631adc63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.511883 1036 tablet_bootstrap.cc:492] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:55.511958 1036 ts_tablet_manager.cc:1367] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:55.512280 1036 raft_consensus.cc:348] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.512348 1036 raft_consensus.cc:374] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.512367 1036 raft_consensus.cc:729] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.512455 1036 consensus_queue.cc:260] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.512686 1036 ts_tablet_manager.cc:1398] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:55.512795 1036 tablet_bootstrap.cc:492] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:55.513777 1036 tablet_bootstrap.cc:654] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.513810 722 consensus_queue.cc:1035] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:55.514510 1041 tablet_bootstrap.cc:492] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:55.515076 1035 tablet_bootstrap.cc:492] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:55.515194 1036 tablet_bootstrap.cc:492] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:55.515277 1036 ts_tablet_manager.cc:1367] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:55.515692 1036 raft_consensus.cc:348] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.515797 1036 raft_consensus.cc:374] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.515836 1036 raft_consensus.cc:729] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.515996 1036 consensus_queue.cc:260] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.516352 1036 ts_tablet_manager.cc:1398] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:55.516475 1035 tablet_bootstrap.cc:654] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.517657 1041 tablet_bootstrap.cc:654] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.518215 1035 tablet_bootstrap.cc:492] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:55.518288 817 tablet_bootstrap.cc:492] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:55.518306 1035 ts_tablet_manager.cc:1367] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:55.518381 817 ts_tablet_manager.cc:1367] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240502 01:58:55.518751 1035 raft_consensus.cc:348] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.518771 817 raft_consensus.cc:348] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.518857 817 raft_consensus.cc:374] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.518857 1035 raft_consensus.cc:374] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.518887 817 raft_consensus.cc:729] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.518887 1035 raft_consensus.cc:729] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.519001 817 consensus_queue.cc:260] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.519007 1035 consensus_queue.cc:260] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.519263 817 ts_tablet_manager.cc:1398] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:55.519261 1035 ts_tablet_manager.cc:1398] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:55.519446 817 tablet_bootstrap.cc:492] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:55.519609 1035 tablet_bootstrap.cc:492] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:55.520443 1035 tablet_bootstrap.cc:654] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.520449 817 tablet_bootstrap.cc:654] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.522543 817 tablet_bootstrap.cc:492] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:55.522631 817 ts_tablet_manager.cc:1367] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:55.522763 1035 tablet_bootstrap.cc:492] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:55.522838 1035 ts_tablet_manager.cc:1367] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:55.522970 817 raft_consensus.cc:348] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.523058 817 raft_consensus.cc:374] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.523085 817 raft_consensus.cc:729] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.523169 817 consensus_queue.cc:260] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.523216 1035 raft_consensus.cc:348] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.523298 1035 raft_consensus.cc:374] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.523289 722 consensus_queue.cc:1035] T b54e83b4026f49408d3f45da92da84e2 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:55.523325 1035 raft_consensus.cc:729] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.523401 817 ts_tablet_manager.cc:1398] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:55.524127 1035 consensus_queue.cc:260] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.524385 1035 ts_tablet_manager.cc:1398] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:55.546242 1041 tablet_bootstrap.cc:492] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:55.546326 1041 ts_tablet_manager.cc:1367] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240502 01:58:55.546726 1041 raft_consensus.cc:348] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.546823 1041 raft_consensus.cc:374] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.546859 1041 raft_consensus.cc:729] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.546989 1041 consensus_queue.cc:260] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.547255 1041 ts_tablet_manager.cc:1398] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:55.547466 1041 tablet_bootstrap.cc:492] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:55.548409 1041 tablet_bootstrap.cc:654] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.549439 1041 tablet_bootstrap.cc:492] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:55.549510 1041 ts_tablet_manager.cc:1367] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:55.549957 1041 raft_consensus.cc:348] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.550081 1041 raft_consensus.cc:374] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.550158 1041 raft_consensus.cc:729] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.550288 1041 consensus_queue.cc:260] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.550565 1041 ts_tablet_manager.cc:1398] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:58:55.552520 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:55.578500 531 raft_consensus.cc:482] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:55.578596 531 raft_consensus.cc:504] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.578958 531 leader_election.cc:288] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:55.579289 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5760b6d6b450d9bc31503aec7e6e4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:55.579336 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5760b6d6b450d9bc31503aec7e6e4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:55.579483 24765 raft_consensus.cc:2454] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:55.579547 24897 raft_consensus.cc:2454] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:55.579849 27380 leader_election.cc:302] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:55.580039 531 raft_consensus.cc:2781] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:55.580097 531 raft_consensus.cc:482] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:55.580133 531 raft_consensus.cc:3037] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.580929 531 raft_consensus.cc:504] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.581251 531 leader_election.cc:288] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:55.581559 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5760b6d6b450d9bc31503aec7e6e4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:55.581666 24897 raft_consensus.cc:3037] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.581651 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5760b6d6b450d9bc31503aec7e6e4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:55.581765 24765 raft_consensus.cc:3037] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.582688 24897 raft_consensus.cc:2454] T 33a5760b6d6b450d9bc31503aec7e6e4 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:55.582831 24765 raft_consensus.cc:2454] T 33a5760b6d6b450d9bc31503aec7e6e4 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:55.583009 27380 leader_election.cc:302] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:55.583175 531 raft_consensus.cc:2781] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:55.583249 531 raft_consensus.cc:686] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:55.583370 531 consensus_queue.cc:237] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.585469 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:55.585872 24483 catalog_manager.cc:5484] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:55.591720 1001 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:55.592056 867 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:55.593154 480 tablet.cc:1722] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:55.598080 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=a11c1219ccf249a0bc158ed83235440c] alter complete (version 3)
W20240502 01:58:55.631335 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:55.631780 868 raft_consensus.cc:1050] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: attempting to promote NON_VOTER bb653087e8554c7591cf547dd969f3be to VOTER
I20240502 01:58:55.632185 868 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:55.633726 24630 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:55.633734 27577 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 LEARNER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:55.634095 27445 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:55.634336 1075 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:55.634569 722 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:55.635035 852 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:55.636696 722 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:55.638209 27445 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:55.639153 24630 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:55.639211 24479 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b reported cstate change: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:55.640332 27577 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb653087e8554c7591cf547dd969f3be (127.17.243.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:55.701395 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:55.701541 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:55.703632 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet c1ab34baf78f47f290cbbc464197e013
I20240502 01:58:55.704326 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab34baf78f47f290cbbc464197e013 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:55 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:55.704381 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab34baf78f47f290cbbc464197e013 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:55 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:55.704679 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet c1ab34baf78f47f290cbbc464197e013 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:55 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:55.704725 1091 tablet_replica.cc:330] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:55.704743 1092 tablet_replica.cc:330] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:55.704879 1092 raft_consensus.cc:2229] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:55.704876 1091 raft_consensus.cc:2229] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:55.705020 1091 raft_consensus.cc:2258] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:55.705034 1093 tablet_replica.cc:330] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:55.705163 1093 raft_consensus.cc:2229] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:55.705282 1093 raft_consensus.cc:2258] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:55.705116 1092 raft_consensus.cc:2258] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:55.705644 1091 ts_tablet_manager.cc:1875] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:55.705837 1093 ts_tablet_manager.cc:1875] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:55.706166 1092 ts_tablet_manager.cc:1875] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:55.711643 1093 ts_tablet_manager.cc:1888] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:55.711648 1091 ts_tablet_manager.cc:1888] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:55.711722 1091 log.cc:1196] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/c1ab34baf78f47f290cbbc464197e013
I20240502 01:58:55.711750 1092 ts_tablet_manager.cc:1888] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:55.711793 1093 log.cc:1196] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/c1ab34baf78f47f290cbbc464197e013
I20240502 01:58:55.711833 1092 log.cc:1196] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/c1ab34baf78f47f290cbbc464197e013
I20240502 01:58:55.712096 1092 ts_tablet_manager.cc:1909] T c1ab34baf78f47f290cbbc464197e013 P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:55.712096 1093 ts_tablet_manager.cc:1909] T c1ab34baf78f47f290cbbc464197e013 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:55.712307 1091 ts_tablet_manager.cc:1909] T c1ab34baf78f47f290cbbc464197e013 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:55.712774 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table_new_name [id=a11c1219ccf249a0bc158ed83235440c]) successfully deleted
I20240502 01:58:55.713040 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table_new_name [id=a11c1219ccf249a0bc158ed83235440c]) successfully deleted
I20240502 01:58:55.713191 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet c1ab34baf78f47f290cbbc464197e013 (table rebalancer_dynamic_table_new_name [id=a11c1219ccf249a0bc158ed83235440c]) successfully deleted
I20240502 01:58:55.762965 815 raft_consensus.cc:482] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:55.763087 815 raft_consensus.cc:504] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.763481 815 leader_election.cc:288] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:55.763732 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deeb5586ff92483a9b85dc8631adc63c" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:55.763859 24897 raft_consensus.cc:2454] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:55.763954 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deeb5586ff92483a9b85dc8631adc63c" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:55.764107 24765 raft_consensus.cc:2454] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:55.764125 24966 leader_election.cc:302] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:55.764529 815 raft_consensus.cc:2781] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:55.764590 815 raft_consensus.cc:482] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:55.764631 815 raft_consensus.cc:3037] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.765611 815 raft_consensus.cc:504] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.765910 815 leader_election.cc:288] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:55.766408 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deeb5586ff92483a9b85dc8631adc63c" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:55.766592 24765 raft_consensus.cc:3037] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.766897 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deeb5586ff92483a9b85dc8631adc63c" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:55.767010 24897 raft_consensus.cc:3037] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.767652 24765 raft_consensus.cc:2454] T deeb5586ff92483a9b85dc8631adc63c P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:55.767899 24966 leader_election.cc:302] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:55.767931 24897 raft_consensus.cc:2454] T deeb5586ff92483a9b85dc8631adc63c P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:55.768307 815 raft_consensus.cc:2781] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:55.768481 815 raft_consensus.cc:686] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:55.768626 815 consensus_queue.cc:237] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:55.770864 24479 catalog_manager.cc:5484] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:55.787523 531 raft_consensus.cc:482] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:55.787631 531 raft_consensus.cc:504] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.788003 531 leader_election.cc:288] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:55.788277 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30939882dceb4f408b5999843644105c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:55.788329 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30939882dceb4f408b5999843644105c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:55.788436 25031 raft_consensus.cc:2454] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:55.788442 27177 raft_consensus.cc:2454] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:55.788702 27379 leader_election.cc:302] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:55.788861 531 raft_consensus.cc:2781] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:55.788918 531 raft_consensus.cc:482] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:55.788954 531 raft_consensus.cc:3037] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.789860 531 raft_consensus.cc:504] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.790222 531 leader_election.cc:288] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:55.790526 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30939882dceb4f408b5999843644105c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:55.790616 25031 raft_consensus.cc:3037] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.790791 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30939882dceb4f408b5999843644105c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:55.790891 27177 raft_consensus.cc:3037] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.791494 25031 raft_consensus.cc:2454] T 30939882dceb4f408b5999843644105c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:55.791582 27177 raft_consensus.cc:2454] T 30939882dceb4f408b5999843644105c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:55.791786 27377 leader_election.cc:302] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:55.791946 531 raft_consensus.cc:2781] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:55.792037 531 raft_consensus.cc:686] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:55.792158 531 consensus_queue.cc:237] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:55.794118 24479 catalog_manager.cc:5484] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:55.805189 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:55.810487 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 83a3ceccb19945c2997096a4d9b4b08b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a]), partition=
I20240502 01:58:55.810714 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a3ceccb19945c2997096a4d9b4b08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.812728 1036 tablet_bootstrap.cc:492] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:55.813082 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 83a3ceccb19945c2997096a4d9b4b08b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a]), partition=
I20240502 01:58:55.813251 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 83a3ceccb19945c2997096a4d9b4b08b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a]), partition=
I20240502 01:58:55.813352 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a3ceccb19945c2997096a4d9b4b08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.813473 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a3ceccb19945c2997096a4d9b4b08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:55.813683 1036 tablet_bootstrap.cc:654] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.814810 1036 tablet_bootstrap.cc:492] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:55.814888 1036 ts_tablet_manager.cc:1367] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:55.815289 1036 raft_consensus.cc:348] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.815392 1036 raft_consensus.cc:374] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.815384 1098 tablet_bootstrap.cc:492] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:55.815424 1036 raft_consensus.cc:729] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.815548 1036 consensus_queue.cc:260] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.815843 1036 ts_tablet_manager.cc:1398] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:55.815934 680 tablet_bootstrap.cc:492] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:55.816790 680 tablet_bootstrap.cc:654] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.816960 1098 tablet_bootstrap.cc:654] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:55.818100 680 tablet_bootstrap.cc:492] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:55.818181 680 ts_tablet_manager.cc:1367] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:55.818615 1098 tablet_bootstrap.cc:492] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:55.818603 680 raft_consensus.cc:348] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.818686 1098 ts_tablet_manager.cc:1367] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:55.818696 680 raft_consensus.cc:374] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.818725 680 raft_consensus.cc:729] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.818833 680 consensus_queue.cc:260] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.819111 680 ts_tablet_manager.cc:1398] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:55.819084 1098 raft_consensus.cc:348] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.819173 1098 raft_consensus.cc:374] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:55.819293 1098 raft_consensus.cc:729] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:55.819461 1098 consensus_queue.cc:260] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.819705 1098 ts_tablet_manager.cc:1398] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:55.864408 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50164
I20240502 01:58:55.864538 24897 raft_consensus.cc:595] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:55.907955 974 raft_consensus.cc:482] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:55.908078 974 raft_consensus.cc:504] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.908519 974 leader_election.cc:288] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:55.908812 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a3ceccb19945c2997096a4d9b4b08b" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:55.908841 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a3ceccb19945c2997096a4d9b4b08b" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:55.908944 24765 raft_consensus.cc:2454] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:55.908944 27577 raft_consensus.cc:2454] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:55.909209 27113 leader_election.cc:302] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:55.909433 974 raft_consensus.cc:2781] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:55.909482 974 raft_consensus.cc:482] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:55.909508 974 raft_consensus.cc:3037] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.910389 974 raft_consensus.cc:504] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.910686 974 leader_election.cc:288] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:55.911190 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a3ceccb19945c2997096a4d9b4b08b" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:55.911212 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a3ceccb19945c2997096a4d9b4b08b" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:55.911307 24765 raft_consensus.cc:3037] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.911322 27577 raft_consensus.cc:3037] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:55.912029 27577 raft_consensus.cc:2454] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:55.912068 24765 raft_consensus.cc:2454] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:55.912304 27113 leader_election.cc:302] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:55.912475 974 raft_consensus.cc:2781] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:55.912565 974 raft_consensus.cc:686] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:55.912698 974 consensus_queue.cc:237] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:55.914626 24486 catalog_manager.cc:5484] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:55.915810 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:55.936906 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet da60bdbf6831421e9f9abec53c6e1d35: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:58:55.957783 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:55.962671 991 consensus_queue.cc:1035] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:55.965816 991 consensus_queue.cc:1035] T 33a5760b6d6b450d9bc31503aec7e6e4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.032522 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:56.036530 27577 raft_consensus.cc:1261] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:56.036628 24765 raft_consensus.cc:1261] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:56.036988 851 consensus_queue.cc:1035] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.037151 1139 consensus_queue.cc:1035] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.039875 24486 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 83a3ceccb19945c2997096a4d9b4b08b (table rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a]). Expected version 1 got 0
I20240502 01:58:56.040635 24479 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 83a3ceccb19945c2997096a4d9b4b08b (table rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a]). Expected version 1 got 0
I20240502 01:58:56.040721 480 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:56.041043 480 tablet.cc:1620] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:56.042201 1001 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:56.042459 1001 tablet.cc:1620] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:56.044216 1146 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:56.044553 1146 tablet.cc:1620] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:56.045349 24482 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 83a3ceccb19945c2997096a4d9b4b08b (table rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a]). Expected version 1 got 0
I20240502 01:58:56.046439 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a] alter complete (version 1)
W20240502 01:58:56.080891 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:56.103652 24897 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84"
dest_uuid: "f46bcd2bc124471981349b1e2726253b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50174
I20240502 01:58:56.103794 24897 raft_consensus.cc:595] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:56.146126 852 raft_consensus.cc:979] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: : Instructing follower bb653087e8554c7591cf547dd969f3be to start an election
I20240502 01:58:56.146234 852 raft_consensus.cc:1067] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Signalling peer bb653087e8554c7591cf547dd969f3be to start an election
I20240502 01:58:56.146651 27577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84"
dest_uuid: "bb653087e8554c7591cf547dd969f3be"
from {username='slave'} at 127.17.243.3:43389
I20240502 01:58:56.146795 27577 raft_consensus.cc:482] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:56.146855 27577 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.147758 27577 raft_consensus.cc:504] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:56.148218 27577 leader_election.cc:288] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:56.148512 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:56.148650 24897 raft_consensus.cc:3032] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:56.148698 24897 raft_consensus.cc:729] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:56.148681 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:56.148854 24630 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.148835 24897 consensus_queue.cc:260] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:56.149017 24897 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.149950 24630 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 2.
I20240502 01:58:56.149963 24897 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 2.
I20240502 01:58:56.149905 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:56.150198 27445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93695e2165e142f79293ac14e4f4fb84" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:56.150327 27445 raft_consensus.cc:3037] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.151383 27445 raft_consensus.cc:2454] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 2.
I20240502 01:58:56.151842 27514 leader_election.cc:302] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:56.151996 1034 raft_consensus.cc:2781] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:56.152081 1034 raft_consensus.cc:686] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 2 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:56.152209 1034 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:56.154740 24486 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 1 to 2, leader changed from f46bcd2bc124471981349b1e2726253b (127.17.243.3) to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 2 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:56.155436 480 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:56.155776 480 tablet.cc:1620] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:56.156293 1146 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:56.156431 1001 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:56.156605 1146 tablet.cc:1620] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:56.156713 1001 tablet.cc:1620] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:56.161579 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=d2a71195b3874f65be89ea653751444a] alter complete (version 2)
I20240502 01:58:56.185870 32105 consensus_queue.cc:1035] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.189922 32105 consensus_queue.cc:1035] T 30939882dceb4f408b5999843644105c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.208279 815 consensus_queue.cc:1035] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.211067 1190 consensus_queue.cc:1035] T deeb5586ff92483a9b85dc8631adc63c P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.263832 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:56.268828 480 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:56.269311 1146 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:56.269610 1001 tablet.cc:1722] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:56.274221 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=d2a71195b3874f65be89ea653751444a] alter complete (version 3)
I20240502 01:58:56.311106 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:56.317178 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 53b71dbd9b504126b7838d3d194d0184 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.317185 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 53b71dbd9b504126b7838d3d194d0184 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.317497 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b71dbd9b504126b7838d3d194d0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.317502 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b71dbd9b504126b7838d3d194d0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.317670 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 53b71dbd9b504126b7838d3d194d0184 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.317976 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b71dbd9b504126b7838d3d194d0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.320080 1192 tablet_bootstrap.cc:492] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:56.320397 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0ec1d9c4a44ca6b825b67ad87484d6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.320454 24743 tablet_service.cc:1467] Processing CreateTablet for tablet b99331513b494137b7b0041a8bfea3e3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.320652 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99331513b494137b7b0041a8bfea3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.320657 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0ec1d9c4a44ca6b825b67ad87484d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.321108 1192 tablet_bootstrap.cc:654] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.321553 24877 tablet_service.cc:1467] Processing CreateTablet for tablet b99331513b494137b7b0041a8bfea3e3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.321792 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99331513b494137b7b0041a8bfea3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.322783 1195 tablet_bootstrap.cc:492] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:56.322829 1098 tablet_bootstrap.cc:492] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:56.323738 1098 tablet_bootstrap.cc:654] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.323855 1196 tablet_bootstrap.cc:492] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:56.323998 1195 tablet_bootstrap.cc:654] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.324350 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0ec1d9c4a44ca6b825b67ad87484d6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.324573 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0ec1d9c4a44ca6b825b67ad87484d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.324856 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0ec1d9c4a44ca6b825b67ad87484d6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.325062 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0ec1d9c4a44ca6b825b67ad87484d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.325300 27157 tablet_service.cc:1467] Processing CreateTablet for tablet b99331513b494137b7b0041a8bfea3e3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d4059c84c2a64cf9830e0fa17ace66a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:56.325567 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99331513b494137b7b0041a8bfea3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.327180 1195 tablet_bootstrap.cc:492] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:56.327175 1098 tablet_bootstrap.cc:492] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:56.327204 1192 tablet_bootstrap.cc:492] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:56.327250 1098 ts_tablet_manager.cc:1367] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:56.327250 1195 ts_tablet_manager.cc:1367] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:56.327286 1192 ts_tablet_manager.cc:1367] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:56.327673 1098 raft_consensus.cc:348] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.327673 1195 raft_consensus.cc:348] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.327782 1098 raft_consensus.cc:374] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.327782 1195 raft_consensus.cc:374] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.327813 1098 raft_consensus.cc:729] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.327813 1195 raft_consensus.cc:729] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.327801 1192 raft_consensus.cc:348] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.327896 1192 raft_consensus.cc:374] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.327930 1192 raft_consensus.cc:729] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.327925 1098 consensus_queue.cc:260] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.327953 1195 consensus_queue.cc:260] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.328156 1192 consensus_queue.cc:260] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.328210 1098 ts_tablet_manager.cc:1398] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:56.328600 1192 ts_tablet_manager.cc:1398] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:56.328847 1195 ts_tablet_manager.cc:1398] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:56.329281 1193 tablet_bootstrap.cc:492] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:56.330178 1193 tablet_bootstrap.cc:654] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.330816 1199 tablet_bootstrap.cc:492] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:56.332573 1199 tablet_bootstrap.cc:654] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.334225 1192 tablet_bootstrap.cc:492] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:56.335439 1192 tablet_bootstrap.cc:654] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.337246 1196 tablet_bootstrap.cc:654] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:56.337846 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:56.338320 1192 tablet_bootstrap.cc:492] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:56.338675 1192 ts_tablet_manager.cc:1367] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:56.339005 1193 tablet_bootstrap.cc:492] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:56.339092 1193 ts_tablet_manager.cc:1367] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:56.339160 1192 raft_consensus.cc:348] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.339239 1192 raft_consensus.cc:374] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.339258 1192 raft_consensus.cc:729] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.339354 1192 consensus_queue.cc:260] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.339593 1192 ts_tablet_manager.cc:1398] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:56.339546 1193 raft_consensus.cc:348] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.339658 1193 raft_consensus.cc:374] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.339694 1193 raft_consensus.cc:729] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.339804 1193 consensus_queue.cc:260] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.340059 1196 tablet_bootstrap.cc:492] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:56.340072 1193 ts_tablet_manager.cc:1398] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:56.340137 1196 ts_tablet_manager.cc:1367] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:56.340253 1193 tablet_bootstrap.cc:492] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:56.340538 1196 raft_consensus.cc:348] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.340636 1196 raft_consensus.cc:374] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.340665 1196 raft_consensus.cc:729] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.340771 1196 consensus_queue.cc:260] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.341104 1193 tablet_bootstrap.cc:654] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.341181 1196 ts_tablet_manager.cc:1398] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:56.341476 1196 tablet_bootstrap.cc:492] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:56.342368 1193 tablet_bootstrap.cc:492] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:56.342459 1193 ts_tablet_manager.cc:1367] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:56.342876 1196 tablet_bootstrap.cc:654] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.342878 1193 raft_consensus.cc:348] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.342981 1193 raft_consensus.cc:374] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.343084 1193 raft_consensus.cc:729] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.343225 1193 consensus_queue.cc:260] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.343559 1193 ts_tablet_manager.cc:1398] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:56.344316 1196 tablet_bootstrap.cc:492] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:56.344393 1196 ts_tablet_manager.cc:1367] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:56.344789 1196 raft_consensus.cc:348] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.344888 1196 raft_consensus.cc:374] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.344915 1196 raft_consensus.cc:729] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.345026 1196 consensus_queue.cc:260] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.345270 1196 ts_tablet_manager.cc:1398] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:56.354282 1199 tablet_bootstrap.cc:492] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:56.354378 1199 ts_tablet_manager.cc:1367] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 01:58:56.354902 1199 raft_consensus.cc:348] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.355019 1199 raft_consensus.cc:374] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.355057 1199 raft_consensus.cc:729] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.355165 1199 consensus_queue.cc:260] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.355448 1199 ts_tablet_manager.cc:1398] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:56.361071 815 raft_consensus.cc:482] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:56.361151 815 raft_consensus.cc:504] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.361510 815 leader_election.cc:288] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:56.361785 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0ec1d9c4a44ca6b825b67ad87484d6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:56.361914 24897 raft_consensus.cc:2454] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:56.361920 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0ec1d9c4a44ca6b825b67ad87484d6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:56.362079 24630 raft_consensus.cc:2454] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:56.362210 24966 leader_election.cc:302] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:56.362349 815 raft_consensus.cc:2781] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:56.362406 815 raft_consensus.cc:482] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:56.362444 815 raft_consensus.cc:3037] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.363436 815 raft_consensus.cc:504] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.363744 815 leader_election.cc:288] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:56.364140 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0ec1d9c4a44ca6b825b67ad87484d6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:56.364239 24630 raft_consensus.cc:3037] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.364590 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0ec1d9c4a44ca6b825b67ad87484d6" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:56.364681 24897 raft_consensus.cc:3037] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.365222 24630 raft_consensus.cc:2454] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:56.365470 24965 leader_election.cc:302] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:56.365625 815 raft_consensus.cc:2781] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:56.365715 815 raft_consensus.cc:686] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:56.365711 24897 raft_consensus.cc:2454] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:56.365857 815 consensus_queue.cc:237] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:56.368083 24482 catalog_manager.cc:5484] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:56.372653 760 raft_consensus.cc:482] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:56.372750 760 raft_consensus.cc:504] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.373109 760 leader_election.cc:288] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:56.373505 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b71dbd9b504126b7838d3d194d0184" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:56.373484 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b71dbd9b504126b7838d3d194d0184" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:56.373601 24632 raft_consensus.cc:2454] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:56.373603 24765 raft_consensus.cc:2454] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:56.373880 27514 leader_election.cc:302] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:56.374188 760 raft_consensus.cc:2781] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:56.374238 760 raft_consensus.cc:482] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:56.374279 760 raft_consensus.cc:3037] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.375137 760 raft_consensus.cc:504] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.375540 760 leader_election.cc:288] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:56.375850 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b71dbd9b504126b7838d3d194d0184" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:56.375878 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b71dbd9b504126b7838d3d194d0184" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:56.375957 24632 raft_consensus.cc:3037] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.375957 24765 raft_consensus.cc:3037] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.376837 24765 raft_consensus.cc:2454] T 53b71dbd9b504126b7838d3d194d0184 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:56.376837 24632 raft_consensus.cc:2454] T 53b71dbd9b504126b7838d3d194d0184 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:56.377239 27514 leader_election.cc:302] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:56.377274 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:56.377378 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:56.377516 760 raft_consensus.cc:2781] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:56.377599 760 raft_consensus.cc:686] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:56.377709 760 consensus_queue.cc:237] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.379576 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 83a3ceccb19945c2997096a4d9b4b08b
I20240502 01:58:56.380283 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a3ceccb19945c2997096a4d9b4b08b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:56 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:56.380532 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a3ceccb19945c2997096a4d9b4b08b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:56 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:56.380565 1234 tablet_replica.cc:330] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:56.380690 1234 raft_consensus.cc:2229] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:56.380841 1235 tablet_replica.cc:330] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:56.380869 1234 raft_consensus.cc:2258] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:56.381108 1235 raft_consensus.cc:2229] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:56.381111 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a3ceccb19945c2997096a4d9b4b08b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:56 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:56.381305 1235 raft_consensus.cc:2258] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:56.381448 1234 ts_tablet_manager.cc:1875] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:56.381420 24482 catalog_manager.cc:5484] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:56.381963 1235 ts_tablet_manager.cc:1875] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:56.382372 1236 tablet_replica.cc:330] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:56.382553 1236 raft_consensus.cc:2229] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:56.382858 1236 raft_consensus.cc:2258] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:56.383872 1236 ts_tablet_manager.cc:1875] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:56.385067 1235 ts_tablet_manager.cc:1888] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:56.385140 1235 log.cc:1196] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/83a3ceccb19945c2997096a4d9b4b08b
I20240502 01:58:56.385325 1234 ts_tablet_manager.cc:1888] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:56.385434 1234 log.cc:1196] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/83a3ceccb19945c2997096a4d9b4b08b
I20240502 01:58:56.385434 1235 ts_tablet_manager.cc:1909] T 83a3ceccb19945c2997096a4d9b4b08b P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:56.385763 1234 ts_tablet_manager.cc:1909] T 83a3ceccb19945c2997096a4d9b4b08b P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:56.386221 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 83a3ceccb19945c2997096a4d9b4b08b (table rebalancer_dynamic_table_new_name [id=d2a71195b3874f65be89ea653751444a]) successfully deleted
I20240502 01:58:56.386507 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 83a3ceccb19945c2997096a4d9b4b08b (table rebalancer_dynamic_table_new_name [id=d2a71195b3874f65be89ea653751444a]) successfully deleted
I20240502 01:58:56.386529 1236 ts_tablet_manager.cc:1888] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:56.386590 1236 log.cc:1196] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/83a3ceccb19945c2997096a4d9b4b08b
I20240502 01:58:56.386821 1236 ts_tablet_manager.cc:1909] T 83a3ceccb19945c2997096a4d9b4b08b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:56.387388 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 83a3ceccb19945c2997096a4d9b4b08b (table rebalancer_dynamic_table_new_name [id=d2a71195b3874f65be89ea653751444a]) successfully deleted
W20240502 01:58:56.432790 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:56.448765 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:56.480933 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:56.486147 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 2436a445987b4316827c4749e32aebbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b]), partition=
I20240502 01:58:56.486217 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 2436a445987b4316827c4749e32aebbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b]), partition=
I20240502 01:58:56.486310 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 2436a445987b4316827c4749e32aebbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b]), partition=
I20240502 01:58:56.486416 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2436a445987b4316827c4749e32aebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.486418 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2436a445987b4316827c4749e32aebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.486562 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2436a445987b4316827c4749e32aebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:56.488566 1195 tablet_bootstrap.cc:492] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:56.488867 1199 tablet_bootstrap.cc:492] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:56.489368 1098 tablet_bootstrap.cc:492] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:56.489416 1195 tablet_bootstrap.cc:654] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.490020 1199 tablet_bootstrap.cc:654] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.490396 1098 tablet_bootstrap.cc:654] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:56.490438 1195 tablet_bootstrap.cc:492] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:56.490504 1195 ts_tablet_manager.cc:1367] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:56.490917 1195 raft_consensus.cc:348] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.491024 1195 raft_consensus.cc:374] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.491058 1195 raft_consensus.cc:729] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.491180 1195 consensus_queue.cc:260] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.491247 1199 tablet_bootstrap.cc:492] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:56.491331 1199 ts_tablet_manager.cc:1367] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:56.491451 1195 ts_tablet_manager.cc:1398] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:56.491801 1199 raft_consensus.cc:348] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.491891 1199 raft_consensus.cc:374] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.491917 1199 raft_consensus.cc:729] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.492013 1199 consensus_queue.cc:260] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.492261 1199 ts_tablet_manager.cc:1398] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:56.492261 1098 tablet_bootstrap.cc:492] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:56.492342 1098 ts_tablet_manager.cc:1367] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:56.492820 1098 raft_consensus.cc:348] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.492933 1098 raft_consensus.cc:374] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:56.492969 1098 raft_consensus.cc:729] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:56.493081 1098 consensus_queue.cc:260] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.493432 1098 ts_tablet_manager.cc:1398] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:56.498698 1040 raft_consensus.cc:482] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:56.498802 1040 raft_consensus.cc:504] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.499205 1040 leader_election.cc:288] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:56.499529 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99331513b494137b7b0041a8bfea3e3" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:56.499542 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99331513b494137b7b0041a8bfea3e3" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:56.499642 27177 raft_consensus.cc:2454] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:56.499677 24897 raft_consensus.cc:2454] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:56.499914 24700 leader_election.cc:302] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:56.500083 1040 raft_consensus.cc:2781] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:56.500137 1040 raft_consensus.cc:482] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:56.500162 1040 raft_consensus.cc:3037] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.500993 1040 raft_consensus.cc:504] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.501286 1040 leader_election.cc:288] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:56.501595 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99331513b494137b7b0041a8bfea3e3" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:56.501694 27177 raft_consensus.cc:3037] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.501742 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99331513b494137b7b0041a8bfea3e3" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:56.501848 24897 raft_consensus.cc:3037] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.502614 27177 raft_consensus.cc:2454] T b99331513b494137b7b0041a8bfea3e3 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:56.502817 24897 raft_consensus.cc:2454] T b99331513b494137b7b0041a8bfea3e3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:56.502866 24700 leader_election.cc:302] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:56.503031 1040 raft_consensus.cc:2781] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:56.503114 1040 raft_consensus.cc:686] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:56.503213 1040 consensus_queue.cc:237] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:56.505205 24482 catalog_manager.cc:5484] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:56.544157 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:56.550292 24897 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:56.550724 1034 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:58:56.587706 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:56.627329 27445 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:56.628226 760 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:56.630987 24630 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:56.632161 1034 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:56.638370 851 raft_consensus.cc:482] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:56.638484 851 raft_consensus.cc:504] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.638845 851 leader_election.cc:288] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:56.639120 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2436a445987b4316827c4749e32aebbe" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:56.639142 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2436a445987b4316827c4749e32aebbe" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:56.639242 27577 raft_consensus.cc:2454] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:56.639259 25032 raft_consensus.cc:2454] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:56.639480 27113 leader_election.cc:302] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:56.639624 851 raft_consensus.cc:2781] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:56.639678 851 raft_consensus.cc:482] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:56.639715 851 raft_consensus.cc:3037] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.640524 851 raft_consensus.cc:504] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.640815 851 leader_election.cc:288] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:56.641086 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2436a445987b4316827c4749e32aebbe" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:56.641098 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2436a445987b4316827c4749e32aebbe" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:56.641192 25032 raft_consensus.cc:3037] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.641192 27577 raft_consensus.cc:3037] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:56.641871 27577 raft_consensus.cc:2454] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:56.642046 25032 raft_consensus.cc:2454] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:56.642168 27113 leader_election.cc:302] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:56.642346 851 raft_consensus.cc:2781] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:56.642436 851 raft_consensus.cc:686] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:56.642560 851 consensus_queue.cc:237] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:56.644621 24479 catalog_manager.cc:5484] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:56.738306 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:56.778286 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:56.782276 27577 raft_consensus.cc:1261] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:56.782305 25032 raft_consensus.cc:1261] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:56.782666 974 consensus_queue.cc:1035] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.782785 851 consensus_queue.cc:1035] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.785089 1287 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:56.785398 1287 tablet.cc:1620] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:56.785512 24482 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 2436a445987b4316827c4749e32aebbe (table rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b]). Expected version 1 got 0
I20240502 01:58:56.785734 1288 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:56.786048 1288 tablet.cc:1620] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:56.789216 1289 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:56.789294 24479 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 2436a445987b4316827c4749e32aebbe (table rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b]). Expected version 1 got 0
I20240502 01:58:56.789474 1289 tablet.cc:1620] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:56.789948 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b] alter complete (version 1)
I20240502 01:58:56.835878 1282 consensus_queue.cc:1035] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.839716 1190 consensus_queue.cc:1035] T 8b0ec1d9c4a44ca6b825b67ad87484d6 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:56.892233 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240502 01:58:56.893818 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:56.897601 1287 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:56.897964 1287 tablet.cc:1620] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:56.897994 1288 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:56.898276 1288 tablet.cc:1620] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:58:56.900070 1289 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:56.900339 1289 tablet.cc:1620] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:56.908334 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=4dcb4d77c55240029a91d861cb815e9b] alter complete (version 2)
W20240502 01:58:56.955730 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:56.978814 760 consensus_queue.cc:1035] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:56.979271 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:56.981998 1034 consensus_queue.cc:1035] T 53b71dbd9b504126b7838d3d194d0184 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:56.999177 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:57.010242 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:57.015668 1287 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:57.016746 1289 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:57.016841 1040 consensus_queue.cc:1035] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.017956 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:57.020710 1288 tablet.cc:1722] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:57.023130 1040 consensus_queue.cc:1035] T b99331513b494137b7b0041a8bfea3e3 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.027354 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=4dcb4d77c55240029a91d861cb815e9b] alter complete (version 3)
I20240502 01:58:57.027480 27424 tablet_service.cc:1467] Processing CreateTablet for tablet f6148c0ec54b46cc836da67ddad87040 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.027673 24877 tablet_service.cc:1467] Processing CreateTablet for tablet fc3bc2cd6c2345249ac87cb1c7f612da (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.027763 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6148c0ec54b46cc836da67ddad87040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.027909 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3bc2cd6c2345249ac87cb1c7f612da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.027863 27423 tablet_service.cc:1467] Processing CreateTablet for tablet fc3bc2cd6c2345249ac87cb1c7f612da (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.028137 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3bc2cd6c2345249ac87cb1c7f612da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.030189 24875 tablet_service.cc:1467] Processing CreateTablet for tablet f6148c0ec54b46cc836da67ddad87040 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.030437 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6148c0ec54b46cc836da67ddad87040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.031484 24743 tablet_service.cc:1467] Processing CreateTablet for tablet fc3bc2cd6c2345249ac87cb1c7f612da (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.031741 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3bc2cd6c2345249ac87cb1c7f612da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.033147 1330 tablet_bootstrap.cc:492] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:57.034080 1331 tablet_bootstrap.cc:492] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:57.034178 1330 tablet_bootstrap.cc:654] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.034269 27558 tablet_service.cc:1467] Processing CreateTablet for tablet f6148c0ec54b46cc836da67ddad87040 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.034411 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 674426b2579d43a594ac161a0e70f7c1 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.034554 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6148c0ec54b46cc836da67ddad87040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.034646 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674426b2579d43a594ac161a0e70f7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.035002 1331 tablet_bootstrap.cc:654] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.035923 1331 tablet_bootstrap.cc:492] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:57.036011 1331 ts_tablet_manager.cc:1367] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:57.036450 1331 raft_consensus.cc:348] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.036541 1331 raft_consensus.cc:374] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.036561 1331 raft_consensus.cc:729] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.036661 1331 consensus_queue.cc:260] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.037077 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 674426b2579d43a594ac161a0e70f7c1 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.037364 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674426b2579d43a594ac161a0e70f7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.037408 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 674426b2579d43a594ac161a0e70f7c1 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b485f974eb0d446faa24a4a9a60aca72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.037600 1331 ts_tablet_manager.cc:1398] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:57.037662 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674426b2579d43a594ac161a0e70f7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.039815 1336 tablet_bootstrap.cc:492] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:57.042356 1336 tablet_bootstrap.cc:654] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:57.044343 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:57.044797 1330 tablet_bootstrap.cc:492] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:57.044879 1330 ts_tablet_manager.cc:1367] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:58:57.045002 1329 tablet_bootstrap.cc:492] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:57.045334 1336 tablet_bootstrap.cc:492] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:57.045415 1336 ts_tablet_manager.cc:1367] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:57.045352 1330 raft_consensus.cc:348] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.045469 1330 raft_consensus.cc:374] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.045503 1330 raft_consensus.cc:729] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.045610 1330 consensus_queue.cc:260] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.045887 1330 ts_tablet_manager.cc:1398] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:57.045890 1336 raft_consensus.cc:348] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.046032 1336 raft_consensus.cc:374] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.046068 1336 raft_consensus.cc:729] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.046142 1335 tablet_bootstrap.cc:492] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:57.046232 1336 consensus_queue.cc:260] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.046339 1330 tablet_bootstrap.cc:492] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:57.046619 1336 ts_tablet_manager.cc:1398] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:57.046837 1334 tablet_bootstrap.cc:492] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:57.047216 1330 tablet_bootstrap.cc:654] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.044659 1338 tablet_bootstrap.cc:492] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:57.048832 1338 tablet_bootstrap.cc:654] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.048995 1330 tablet_bootstrap.cc:492] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:57.049093 1330 ts_tablet_manager.cc:1367] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:57.049614 1330 raft_consensus.cc:348] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.049727 1330 raft_consensus.cc:374] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.049768 1330 raft_consensus.cc:729] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.049885 1330 consensus_queue.cc:260] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.050217 1330 ts_tablet_manager.cc:1398] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:57.050619 1335 tablet_bootstrap.cc:654] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.051838 1329 tablet_bootstrap.cc:654] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.047756 1334 tablet_bootstrap.cc:654] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.053876 1334 tablet_bootstrap.cc:492] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:57.054015 1334 ts_tablet_manager.cc:1367] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:57.054543 1334 raft_consensus.cc:348] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.055089 1329 tablet_bootstrap.cc:492] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:57.055559 1335 tablet_bootstrap.cc:492] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:57.055660 1335 ts_tablet_manager.cc:1367] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:57.056061 1335 raft_consensus.cc:348] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.056159 1335 raft_consensus.cc:374] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.056185 1335 raft_consensus.cc:729] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.056304 1335 consensus_queue.cc:260] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.056584 1335 ts_tablet_manager.cc:1398] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:57.057137 1338 tablet_bootstrap.cc:492] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:57.057246 1338 ts_tablet_manager.cc:1367] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:58:57.057801 1338 raft_consensus.cc:348] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.057955 1338 raft_consensus.cc:374] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.058003 1338 raft_consensus.cc:729] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.058140 1338 consensus_queue.cc:260] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.058465 1338 ts_tablet_manager.cc:1398] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:57.058601 1329 ts_tablet_manager.cc:1367] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:58:57.059074 1329 raft_consensus.cc:348] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.059196 1329 raft_consensus.cc:374] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.059228 1329 raft_consensus.cc:729] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.059353 1329 consensus_queue.cc:260] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.059605 1334 raft_consensus.cc:374] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.059664 1334 raft_consensus.cc:729] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.059799 1334 consensus_queue.cc:260] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.060111 1334 ts_tablet_manager.cc:1398] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:57.060400 1329 ts_tablet_manager.cc:1398] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:57.060596 1329 tablet_bootstrap.cc:492] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:57.061741 1329 tablet_bootstrap.cc:654] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.063963 1329 tablet_bootstrap.cc:492] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:57.064103 1329 ts_tablet_manager.cc:1367] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:57.064615 1329 raft_consensus.cc:348] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.065711 1329 raft_consensus.cc:374] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.065804 1329 raft_consensus.cc:729] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.065972 1329 consensus_queue.cc:260] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.066303 1329 ts_tablet_manager.cc:1398] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 01:58:57.116315 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:57.133431 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:57.133575 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:57.135612 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 2436a445987b4316827c4749e32aebbe
I20240502 01:58:57.136116 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 2436a445987b4316827c4749e32aebbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:57 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:58:57.136328 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet 2436a445987b4316827c4749e32aebbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:57 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:57.136360 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet 2436a445987b4316827c4749e32aebbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:57 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:57.136621 1378 tablet_replica.cc:330] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:57.136652 1379 tablet_replica.cc:330] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:57.136806 1379 raft_consensus.cc:2229] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:57.136873 1377 tablet_replica.cc:330] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:58:57.136785 1378 raft_consensus.cc:2229] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:57.137014 1377 raft_consensus.cc:2229] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:57.137122 1379 raft_consensus.cc:2258] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.137161 1377 raft_consensus.cc:2258] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.137408 1378 raft_consensus.cc:2258] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.137804 1379 ts_tablet_manager.cc:1875] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:57.137959 1377 ts_tablet_manager.cc:1875] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:57.138108 1378 ts_tablet_manager.cc:1875] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:57.139680 1379 ts_tablet_manager.cc:1888] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:57.139743 1379 log.cc:1196] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/2436a445987b4316827c4749e32aebbe
I20240502 01:58:57.139983 1379 ts_tablet_manager.cc:1909] T 2436a445987b4316827c4749e32aebbe P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:57.140107 1377 ts_tablet_manager.cc:1888] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:57.140182 1377 log.cc:1196] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/2436a445987b4316827c4749e32aebbe
I20240502 01:58:57.140272 1378 ts_tablet_manager.cc:1888] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:57.140338 1378 log.cc:1196] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/2436a445987b4316827c4749e32aebbe
I20240502 01:58:57.140533 1377 ts_tablet_manager.cc:1909] T 2436a445987b4316827c4749e32aebbe P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:58:57.140579 1378 ts_tablet_manager.cc:1909] T 2436a445987b4316827c4749e32aebbe P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:57.140725 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 2436a445987b4316827c4749e32aebbe (table rebalancer_dynamic_table_new_name [id=4dcb4d77c55240029a91d861cb815e9b]) successfully deleted
I20240502 01:58:57.141175 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 2436a445987b4316827c4749e32aebbe (table rebalancer_dynamic_table_new_name [id=4dcb4d77c55240029a91d861cb815e9b]) successfully deleted
I20240502 01:58:57.141263 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 2436a445987b4316827c4749e32aebbe (table rebalancer_dynamic_table_new_name [id=4dcb4d77c55240029a91d861cb815e9b]) successfully deleted
I20240502 01:58:57.162110 32105 raft_consensus.cc:482] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.162223 32105 raft_consensus.cc:504] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.162731 32105 leader_election.cc:288] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:57.163080 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3bc2cd6c2345249ac87cb1c7f612da" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:57.163092 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3bc2cd6c2345249ac87cb1c7f612da" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:57.163211 24896 raft_consensus.cc:2454] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:57.163211 24765 raft_consensus.cc:2454] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:57.163486 27380 leader_election.cc:302] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:57.163713 32105 raft_consensus.cc:2781] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.163769 32105 raft_consensus.cc:482] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.163806 32105 raft_consensus.cc:3037] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.164839 32105 raft_consensus.cc:504] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.165179 32105 leader_election.cc:288] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:57.165449 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3bc2cd6c2345249ac87cb1c7f612da" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:57.165544 24896 raft_consensus.cc:3037] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.166497 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3bc2cd6c2345249ac87cb1c7f612da" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:57.166620 24765 raft_consensus.cc:3037] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.166740 24896 raft_consensus.cc:2454] T fc3bc2cd6c2345249ac87cb1c7f612da P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:57.167073 27380 leader_election.cc:302] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:57.167533 24765 raft_consensus.cc:2454] T fc3bc2cd6c2345249ac87cb1c7f612da P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:57.167886 32105 raft_consensus.cc:2781] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.167989 32105 raft_consensus.cc:686] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:57.168107 32105 consensus_queue.cc:237] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.170054 24486 catalog_manager.cc:5484] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:57.174252 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.214251 27577 consensus_queue.cc:237] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } }
I20240502 01:58:57.215433 27445 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:57.215835 1034 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:57.215999 24632 raft_consensus.cc:1261] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Refusing update from remote peer bb653087e8554c7591cf547dd969f3be: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:57.216625 1034 consensus_queue.cc:1035] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:57.218209 1278 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:57.218637 24632 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 588bfc090a5247598cd5d3cee5ad2007 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:57.219799 24474 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93695e2165e142f79293ac14e4f4fb84 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:57.220886 27445 raft_consensus.cc:2932] T 93695e2165e142f79293ac14e4f4fb84 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } } }
I20240502 01:58:57.221032 24486 catalog_manager.cc:5484] T 93695e2165e142f79293ac14e4f4fb84 P bb653087e8554c7591cf547dd969f3be reported cstate change: config changed from index 8 to 10, VOTER f46bcd2bc124471981349b1e2726253b (127.17.243.3) evicted. New cstate: current_term: 2 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:57.224500 24875 tablet_service.cc:1514] Processing DeleteTablet for tablet 93695e2165e142f79293ac14e4f4fb84 with delete_type TABLET_DATA_TOMBSTONED (TS f46bcd2bc124471981349b1e2726253b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:57.224854 1381 tablet_replica.cc:330] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:57.225015 1381 raft_consensus.cc:2229] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:57.225147 1381 raft_consensus.cc:2258] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.225798 1381 ts_tablet_manager.cc:1875] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:57.227747 1381 ts_tablet_manager.cc:1888] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:57.227821 1381 log.cc:1196] T 93695e2165e142f79293ac14e4f4fb84 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/93695e2165e142f79293ac14e4f4fb84
I20240502 01:58:57.228281 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 93695e2165e142f79293ac14e4f4fb84 (table rebalance_test_table_17 [id=67f274ce6fca4a3ba9cdbb22e5ecbd3c]) successfully deleted
I20240502 01:58:57.237124 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:57.242725 24743 tablet_service.cc:1467] Processing CreateTablet for tablet b57afc4020f24ebf8a0aecf6aea9320d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8]), partition=
W20240502 01:58:57.242928 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.242972 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57afc4020f24ebf8a0aecf6aea9320d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.243067 27157 tablet_service.cc:1467] Processing CreateTablet for tablet b57afc4020f24ebf8a0aecf6aea9320d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8]), partition=
I20240502 01:58:57.243328 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57afc4020f24ebf8a0aecf6aea9320d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.244845 1331 tablet_bootstrap.cc:492] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:57.245031 27558 tablet_service.cc:1467] Processing CreateTablet for tablet b57afc4020f24ebf8a0aecf6aea9320d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8]), partition=
I20240502 01:58:57.245206 1336 tablet_bootstrap.cc:492] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:57.245249 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57afc4020f24ebf8a0aecf6aea9320d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.245728 1331 tablet_bootstrap.cc:654] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.245950 1336 tablet_bootstrap.cc:654] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.246843 1331 tablet_bootstrap.cc:492] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:57.246948 1331 ts_tablet_manager.cc:1367] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:57.247323 1334 tablet_bootstrap.cc:492] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:57.247395 1331 raft_consensus.cc:348] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.247489 1331 raft_consensus.cc:374] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.247516 1331 raft_consensus.cc:729] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.247622 1331 consensus_queue.cc:260] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.247913 1331 ts_tablet_manager.cc:1398] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:57.248265 1334 tablet_bootstrap.cc:654] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.248339 1336 tablet_bootstrap.cc:492] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:57.248409 1336 ts_tablet_manager.cc:1367] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:57.249445 1336 raft_consensus.cc:348] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.249564 1336 raft_consensus.cc:374] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.249610 1336 raft_consensus.cc:729] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.249733 1336 consensus_queue.cc:260] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.250048 1336 ts_tablet_manager.cc:1398] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:57.251892 1334 tablet_bootstrap.cc:492] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:57.251986 1334 ts_tablet_manager.cc:1367] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:57.252421 1334 raft_consensus.cc:348] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.252528 1334 raft_consensus.cc:374] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.252568 1334 raft_consensus.cc:729] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.252689 1334 consensus_queue.cc:260] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.253618 1334 ts_tablet_manager.cc:1398] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:57.257025 32105 raft_consensus.cc:482] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.257114 32105 raft_consensus.cc:504] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.257513 32105 leader_election.cc:288] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:57.257772 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6148c0ec54b46cc836da67ddad87040" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:57.257828 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6148c0ec54b46cc836da67ddad87040" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:57.257890 27577 raft_consensus.cc:2454] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:57.257978 24896 raft_consensus.cc:2454] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:57.258168 27380 leader_election.cc:302] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:57.258440 32105 raft_consensus.cc:2781] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.258491 32105 raft_consensus.cc:482] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.258531 32105 raft_consensus.cc:3037] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.259280 32105 raft_consensus.cc:504] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.259699 32105 leader_election.cc:288] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:57.260032 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6148c0ec54b46cc836da67ddad87040" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:57.260039 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6148c0ec54b46cc836da67ddad87040" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:57.260123 27577 raft_consensus.cc:3037] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.260134 24896 raft_consensus.cc:3037] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.260811 27577 raft_consensus.cc:2454] T f6148c0ec54b46cc836da67ddad87040 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:57.261018 24896 raft_consensus.cc:2454] T f6148c0ec54b46cc836da67ddad87040 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:57.261325 27380 leader_election.cc:302] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:57.261499 32105 raft_consensus.cc:2781] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.261592 32105 raft_consensus.cc:686] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:57.261716 32105 consensus_queue.cc:237] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.263598 24486 catalog_manager.cc:5484] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:57.281905 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:58:57.289611 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35596116445411f857f52eadd963a16: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:57.341533 1139 raft_consensus.cc:482] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.341645 1139 raft_consensus.cc:504] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.342137 1139 leader_election.cc:288] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:57.342473 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674426b2579d43a594ac161a0e70f7c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:57.342619 25032 raft_consensus.cc:2454] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:57.342731 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674426b2579d43a594ac161a0e70f7c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:57.342877 24632 raft_consensus.cc:2454] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:57.343355 27112 leader_election.cc:302] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:57.343534 1139 raft_consensus.cc:2781] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.343581 1139 raft_consensus.cc:482] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.343631 1139 raft_consensus.cc:3037] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.344728 1139 raft_consensus.cc:504] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.345031 1139 leader_election.cc:288] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:57.345408 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674426b2579d43a594ac161a0e70f7c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:57.345527 25032 raft_consensus.cc:3037] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.345990 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674426b2579d43a594ac161a0e70f7c1" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:57.346096 24632 raft_consensus.cc:3037] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.346666 25032 raft_consensus.cc:2454] T 674426b2579d43a594ac161a0e70f7c1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:57.346843 24632 raft_consensus.cc:2454] T 674426b2579d43a594ac161a0e70f7c1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:57.346947 27110 leader_election.cc:302] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:57.347101 1139 raft_consensus.cc:2781] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.347203 1139 raft_consensus.cc:686] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:57.347364 1139 consensus_queue.cc:237] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.349789 24483 catalog_manager.cc:5484] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:57.374692 1326 raft_consensus.cc:482] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.374795 1326 raft_consensus.cc:504] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.375238 1326 leader_election.cc:288] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:57.375566 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57afc4020f24ebf8a0aecf6aea9320d" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:57.375547 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57afc4020f24ebf8a0aecf6aea9320d" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:57.375692 27177 raft_consensus.cc:2454] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:57.375699 27577 raft_consensus.cc:2454] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:57.375967 24700 leader_election.cc:302] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:57.376253 1326 raft_consensus.cc:2781] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.376320 1326 raft_consensus.cc:482] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.376369 1326 raft_consensus.cc:3037] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.377209 1326 raft_consensus.cc:504] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.377496 1326 leader_election.cc:288] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:57.377813 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57afc4020f24ebf8a0aecf6aea9320d" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:57.377909 27577 raft_consensus.cc:3037] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.378780 27577 raft_consensus.cc:2454] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:57.379084 24701 leader_election.cc:302] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:57.379213 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57afc4020f24ebf8a0aecf6aea9320d" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:57.379303 1326 raft_consensus.cc:2781] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.379319 27177 raft_consensus.cc:3037] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.379390 1326 raft_consensus.cc:686] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:57.379519 1326 consensus_queue.cc:237] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:57.380292 27177 raft_consensus.cc:2454] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:57.381668 24483 catalog_manager.cc:5484] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:57.422724 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.427773 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.504401 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:57.508157 27177 raft_consensus.cc:1261] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:57.508239 27577 raft_consensus.cc:1261] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:57.508622 1445 consensus_queue.cc:1035] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.508775 1326 consensus_queue.cc:1035] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.511801 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet b57afc4020f24ebf8a0aecf6aea9320d (table rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8]). Expected version 1 got 0
I20240502 01:58:57.511826 1448 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:57.512168 1448 tablet.cc:1620] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:57.513182 24486 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet b57afc4020f24ebf8a0aecf6aea9320d (table rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8]). Expected version 1 got 0
I20240502 01:58:57.515447 1287 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:57.515597 1289 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:57.515740 1287 tablet.cc:1620] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:57.515838 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8] alter complete (version 1)
I20240502 01:58:57.515866 1289 tablet.cc:1620] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:57.517462 24483 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet b57afc4020f24ebf8a0aecf6aea9320d (table rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8]). Expected version 1 got 0
W20240502 01:58:57.531906 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.547812 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:57.562006 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.571839 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.593398 991 consensus_queue.cc:1035] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.601531 991 consensus_queue.cc:1035] T fc3bc2cd6c2345249ac87cb1c7f612da P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:57.614486 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.621559 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240502 01:58:57.626950 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.627722 1448 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:57.628022 1448 tablet.cc:1620] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:57.628264 1287 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:57.628578 1287 tablet.cc:1620] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:57.628551 1289 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:57.628839 1289 tablet.cc:1620] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:58:57.633533 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=3f05db33a0904b8ba4042e4bae433fb8] alter complete (version 2)
I20240502 01:58:57.640697 991 consensus_queue.cc:1035] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.643746 1481 consensus_queue.cc:1035] T f6148c0ec54b46cc836da67ddad87040 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:57.651039 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.706813 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:58:57.726014 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.736354 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:57.741514 851 consensus_queue.cc:1035] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:57.742606 1448 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:57.744391 1287 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:57.745143 1289 tablet.cc:1722] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:57.749852 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=3f05db33a0904b8ba4042e4bae433fb8] alter complete (version 3)
I20240502 01:58:57.750269 974 consensus_queue.cc:1035] T 674426b2579d43a594ac161a0e70f7c1 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:57.765254 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.768455 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.778249 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:57.819418 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.851151 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:57.851289 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:57.853267 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet b57afc4020f24ebf8a0aecf6aea9320d
I20240502 01:58:57.853772 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet b57afc4020f24ebf8a0aecf6aea9320d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:57 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:58:57.853832 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet b57afc4020f24ebf8a0aecf6aea9320d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:57 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:57.854023 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet b57afc4020f24ebf8a0aecf6aea9320d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:57 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:57.854291 1513 tablet_replica.cc:330] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:57.854404 1514 tablet_replica.cc:330] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:57.854429 1512 tablet_replica.cc:330] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:58:57.854460 1513 raft_consensus.cc:2229] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:57.854557 1514 raft_consensus.cc:2229] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:57.854568 1512 raft_consensus.cc:2229] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:57.854689 1512 raft_consensus.cc:2258] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.854710 1514 raft_consensus.cc:2258] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.854714 1513 raft_consensus.cc:2258] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.855319 1514 ts_tablet_manager.cc:1875] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:57.855319 1512 ts_tablet_manager.cc:1875] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:57.855744 1513 ts_tablet_manager.cc:1875] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:57.858017 1512 ts_tablet_manager.cc:1888] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W20240502 01:58:57.858026 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.858054 1513 ts_tablet_manager.cc:1888] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:57.858119 1513 log.cc:1196] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/b57afc4020f24ebf8a0aecf6aea9320d
I20240502 01:58:57.858139 1512 log.cc:1196] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/b57afc4020f24ebf8a0aecf6aea9320d
I20240502 01:58:57.858388 1512 ts_tablet_manager.cc:1909] T b57afc4020f24ebf8a0aecf6aea9320d P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:58:57.858476 1513 ts_tablet_manager.cc:1909] T b57afc4020f24ebf8a0aecf6aea9320d P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:57.859015 1514 ts_tablet_manager.cc:1888] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:57.859146 1514 log.cc:1196] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/b57afc4020f24ebf8a0aecf6aea9320d
I20240502 01:58:57.859124 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet b57afc4020f24ebf8a0aecf6aea9320d (table rebalancer_dynamic_table_new_name [id=3f05db33a0904b8ba4042e4bae433fb8]) successfully deleted
I20240502 01:58:57.859268 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet b57afc4020f24ebf8a0aecf6aea9320d (table rebalancer_dynamic_table_new_name [id=3f05db33a0904b8ba4042e4bae433fb8]) successfully deleted
I20240502 01:58:57.859520 1514 ts_tablet_manager.cc:1909] T b57afc4020f24ebf8a0aecf6aea9320d P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:57.860177 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet b57afc4020f24ebf8a0aecf6aea9320d (table rebalancer_dynamic_table_new_name [id=3f05db33a0904b8ba4042e4bae433fb8]) successfully deleted
I20240502 01:58:57.904778 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:58:57.908193 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.911154 24610 tablet_service.cc:1467] Processing CreateTablet for tablet d0c09a31b1ba43639e1290bca14eefad (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.911388 27558 tablet_service.cc:1467] Processing CreateTablet for tablet d0c09a31b1ba43639e1290bca14eefad (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.911464 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c09a31b1ba43639e1290bca14eefad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.911628 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c09a31b1ba43639e1290bca14eefad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.912544 25011 tablet_service.cc:1467] Processing CreateTablet for tablet bd8faf173a6e43d5be01ce66ff16e5c5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.912760 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8faf173a6e43d5be01ce66ff16e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.913800 1516 tablet_bootstrap.cc:492] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:57.913987 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 65e9454c295a4a0cac4b955b916cd547 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.914222 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e9454c295a4a0cac4b955b916cd547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.915089 1516 tablet_bootstrap.cc:654] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.915710 1515 tablet_bootstrap.cc:492] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:57.916370 1519 tablet_bootstrap.cc:492] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:57.916350 24743 tablet_service.cc:1467] Processing CreateTablet for tablet bd8faf173a6e43d5be01ce66ff16e5c5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.916564 1515 tablet_bootstrap.cc:654] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.916606 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8faf173a6e43d5be01ce66ff16e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.917174 1519 tablet_bootstrap.cc:654] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.917482 1515 tablet_bootstrap.cc:492] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:57.917549 1515 ts_tablet_manager.cc:1367] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:57.917976 1515 raft_consensus.cc:348] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.918071 1515 raft_consensus.cc:374] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.918094 1515 raft_consensus.cc:729] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.918197 1515 consensus_queue.cc:260] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.918452 1515 ts_tablet_manager.cc:1398] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:57.918870 25012 tablet_service.cc:1467] Processing CreateTablet for tablet d0c09a31b1ba43639e1290bca14eefad (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.919095 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c09a31b1ba43639e1290bca14eefad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.921114 25010 tablet_service.cc:1467] Processing CreateTablet for tablet 65e9454c295a4a0cac4b955b916cd547 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.921326 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e9454c295a4a0cac4b955b916cd547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.923204 27423 tablet_service.cc:1467] Processing CreateTablet for tablet bd8faf173a6e43d5be01ce66ff16e5c5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.923403 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8faf173a6e43d5be01ce66ff16e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.925477 1523 tablet_bootstrap.cc:492] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:57.926560 1523 tablet_bootstrap.cc:654] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.927186 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 65e9454c295a4a0cac4b955b916cd547 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=7981c18818e740808d4554ff32587393]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:57.927378 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e9454c295a4a0cac4b955b916cd547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.929432 1516 tablet_bootstrap.cc:492] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:57.929520 1516 ts_tablet_manager.cc:1367] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:58:57.929702 1522 tablet_bootstrap.cc:492] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:57.929989 1516 raft_consensus.cc:348] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.930094 1516 raft_consensus.cc:374] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.930125 1516 raft_consensus.cc:729] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.930246 1516 consensus_queue.cc:260] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.930524 1516 ts_tablet_manager.cc:1398] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:57.930549 1517 tablet_bootstrap.cc:492] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:57.930824 1516 tablet_bootstrap.cc:492] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:57.931360 1522 tablet_bootstrap.cc:654] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.931684 1516 tablet_bootstrap.cc:654] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.932214 1517 tablet_bootstrap.cc:654] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.932870 1516 tablet_bootstrap.cc:492] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:57.932998 1516 ts_tablet_manager.cc:1367] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:57.933265 1517 tablet_bootstrap.cc:492] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:57.933365 1517 ts_tablet_manager.cc:1367] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:57.933614 1516 raft_consensus.cc:348] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.933768 1516 raft_consensus.cc:374] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.933844 1516 raft_consensus.cc:729] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.933841 1517 raft_consensus.cc:348] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.933960 1517 raft_consensus.cc:374] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.933998 1517 raft_consensus.cc:729] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.934113 1517 consensus_queue.cc:260] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.934183 1516 consensus_queue.cc:260] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.934387 1517 ts_tablet_manager.cc:1398] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:57.934417 1523 tablet_bootstrap.cc:492] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:57.934566 1517 tablet_bootstrap.cc:492] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:57.934718 1516 ts_tablet_manager.cc:1398] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:57.935067 1519 tablet_bootstrap.cc:492] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:57.935288 1519 ts_tablet_manager.cc:1367] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 01:58:57.935400 1517 tablet_bootstrap.cc:654] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.935909 1280 raft_consensus.cc:482] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.935972 1523 ts_tablet_manager.cc:1367] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 01:58:57.935981 1280 raft_consensus.cc:504] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.936386 1517 tablet_bootstrap.cc:492] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:57.936467 1517 ts_tablet_manager.cc:1367] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:57.936475 1280 leader_election.cc:288] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:57.936416 1523 raft_consensus.cc:348] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.936771 1523 raft_consensus.cc:374] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.936841 1523 raft_consensus.cc:729] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.936858 1517 raft_consensus.cc:348] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.936954 1517 raft_consensus.cc:374] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.936992 1517 raft_consensus.cc:729] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.937063 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c09a31b1ba43639e1290bca14eefad" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:57.937096 1517 consensus_queue.cc:260] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.937095 1523 consensus_queue.cc:260] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.937193 25032 raft_consensus.cc:2454] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:57.937382 1523 ts_tablet_manager.cc:1398] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:57.937449 1517 ts_tablet_manager.cc:1398] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:57.937587 1517 tablet_bootstrap.cc:492] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:57.937870 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c09a31b1ba43639e1290bca14eefad" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:57.938027 27577 raft_consensus.cc:2454] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:57.938378 1517 tablet_bootstrap.cc:654] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.938529 24567 leader_election.cc:302] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:57.938675 1280 raft_consensus.cc:2781] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.938717 1280 raft_consensus.cc:482] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.938750 1280 raft_consensus.cc:3037] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.939637 1522 tablet_bootstrap.cc:492] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:57.939731 1522 ts_tablet_manager.cc:1367] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:58:57.939698 1280 raft_consensus.cc:504] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.935820 1519 raft_consensus.cc:348] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.940006 1280 leader_election.cc:288] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:58:57.940016 1519 raft_consensus.cc:374] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.940059 1519 raft_consensus.cc:729] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.940178 1519 consensus_queue.cc:260] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.940186 1522 raft_consensus.cc:348] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.940304 1522 raft_consensus.cc:374] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.940340 1522 raft_consensus.cc:729] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.940450 1522 consensus_queue.cc:260] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.940500 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c09a31b1ba43639e1290bca14eefad" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:57.940582 27577 raft_consensus.cc:3037] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.940735 1522 ts_tablet_manager.cc:1398] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:57.941151 1519 ts_tablet_manager.cc:1398] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:57.941520 27577 raft_consensus.cc:2454] T d0c09a31b1ba43639e1290bca14eefad P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:57.941810 1517 tablet_bootstrap.cc:492] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:57.941895 1517 ts_tablet_manager.cc:1367] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:57.942020 1376 raft_consensus.cc:482] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.942106 1376 raft_consensus.cc:504] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.942328 1517 raft_consensus.cc:348] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.942415 1517 raft_consensus.cc:374] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.942445 1517 raft_consensus.cc:729] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.942437 1376 leader_election.cc:288] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:57.942543 1517 consensus_queue.cc:260] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.942770 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9454c295a4a0cac4b955b916cd547" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:57.942885 25031 raft_consensus.cc:2454] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:57.943271 24830 leader_election.cc:302] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:57.943287 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9454c295a4a0cac4b955b916cd547" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:57.943390 24632 raft_consensus.cc:2454] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:57.944304 24567 leader_election.cc:302] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:57.944502 1376 raft_consensus.cc:2781] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.944554 1376 raft_consensus.cc:482] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.944595 1376 raft_consensus.cc:3037] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.945341 1280 raft_consensus.cc:2781] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.945423 1280 raft_consensus.cc:686] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:57.945537 1280 consensus_queue.cc:237] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:57.945586 1376 raft_consensus.cc:504] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.945902 1376 leader_election.cc:288] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:58:57.946338 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9454c295a4a0cac4b955b916cd547" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:57.946434 24632 raft_consensus.cc:3037] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.946413 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9454c295a4a0cac4b955b916cd547" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:57.946503 25032 raft_consensus.cc:3037] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.947273 24632 raft_consensus.cc:2454] T 65e9454c295a4a0cac4b955b916cd547 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:57.947428 25032 raft_consensus.cc:2454] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:57.947618 24832 leader_election.cc:302] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:57.942804 1517 ts_tablet_manager.cc:1398] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:57.948045 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c09a31b1ba43639e1290bca14eefad" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:57.948148 25032 raft_consensus.cc:3037] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.948549 1376 raft_consensus.cc:2781] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.948720 1376 raft_consensus.cc:686] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:57.948859 1376 consensus_queue.cc:237] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:58:57.948937 25032 raft_consensus.cc:2454] T d0c09a31b1ba43639e1290bca14eefad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:57.951422 24486 catalog_manager.cc:5484] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:57.952682 24483 catalog_manager.cc:5484] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:57.954157 24812 tablet.cc:2369] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:57.955768 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:57.960074 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 4966d864897b46a584b0ba863940f731 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]), partition=
I20240502 01:58:57.960091 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 4966d864897b46a584b0ba863940f731 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]), partition=
I20240502 01:58:57.960294 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966d864897b46a584b0ba863940f731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.960304 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966d864897b46a584b0ba863940f731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.961869 1516 tablet_bootstrap.cc:492] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:57.962196 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 4966d864897b46a584b0ba863940f731 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]), partition=
I20240502 01:58:57.962412 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4966d864897b46a584b0ba863940f731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:57.962771 1516 tablet_bootstrap.cc:654] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.963410 1282 raft_consensus.cc:482] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:57.963498 1282 raft_consensus.cc:504] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.963888 1282 leader_election.cc:288] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:57.964229 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8faf173a6e43d5be01ce66ff16e5c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:57.964335 1516 tablet_bootstrap.cc:492] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:57.964362 24765 raft_consensus.cc:2454] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:57.964417 1516 ts_tablet_manager.cc:1367] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:58:57.964655 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8faf173a6e43d5be01ce66ff16e5c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:57.964731 1523 tablet_bootstrap.cc:492] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:57.964738 24966 leader_election.cc:302] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:57.964766 27446 raft_consensus.cc:2454] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:58:57.964830 1516 raft_consensus.cc:348] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.964915 1516 raft_consensus.cc:374] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.964943 1516 raft_consensus.cc:729] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.965044 1516 consensus_queue.cc:260] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.965332 1516 ts_tablet_manager.cc:1398] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:57.965548 1282 raft_consensus.cc:2781] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:57.965611 1282 raft_consensus.cc:482] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:57.965655 1282 raft_consensus.cc:3037] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.965750 1523 tablet_bootstrap.cc:654] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
W20240502 01:58:57.966169 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:57.966573 1533 tablet_bootstrap.cc:492] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:57.966603 1282 raft_consensus.cc:504] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.966910 1282 leader_election.cc:288] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:57.967177 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8faf173a6e43d5be01ce66ff16e5c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:57.967273 27446 raft_consensus.cc:3037] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.967339 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8faf173a6e43d5be01ce66ff16e5c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:57.967439 24765 raft_consensus.cc:3037] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:57.968168 27446 raft_consensus.cc:2454] T bd8faf173a6e43d5be01ce66ff16e5c5 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:57.968313 24765 raft_consensus.cc:2454] T bd8faf173a6e43d5be01ce66ff16e5c5 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:58:57.968614 24966 leader_election.cc:302] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:57.968750 1282 raft_consensus.cc:2781] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:57.968789 1533 tablet_bootstrap.cc:654] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:57.968819 1282 raft_consensus.cc:686] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:58:57.968927 1282 consensus_queue.cc:237] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:57.969177 1523 tablet_bootstrap.cc:492] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:57.969247 1523 ts_tablet_manager.cc:1367] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:57.969666 1523 raft_consensus.cc:348] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.969754 1523 raft_consensus.cc:374] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.969784 1523 raft_consensus.cc:729] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.969906 1523 consensus_queue.cc:260] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.970199 1523 ts_tablet_manager.cc:1398] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:57.971663 1533 tablet_bootstrap.cc:492] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:57.971793 1533 ts_tablet_manager.cc:1367] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:57.972260 1533 raft_consensus.cc:348] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.972272 24486 catalog_manager.cc:5484] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:57.972390 1533 raft_consensus.cc:374] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:57.972460 1533 raft_consensus.cc:729] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:57.972602 1533 consensus_queue.cc:260] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:57.973737 1533 ts_tablet_manager.cc:1398] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 01:58:58.037410 25081 tablet.cc:2369] T 65e9454c295a4a0cac4b955b916cd547 P e0b9e7df4e8543fa9b8e3ffc24330657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:58.055234 24680 tablet.cc:2369] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:58.125082 24946 tablet.cc:2369] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:58.148769 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:58.155797 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:58:58.204366 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:58.304270 1479 raft_consensus.cc:482] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:58.304394 1479 raft_consensus.cc:504] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:58.304848 1479 leader_election.cc:288] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:58.305171 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966d864897b46a584b0ba863940f731" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:58.305198 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966d864897b46a584b0ba863940f731" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:58.305295 27177 raft_consensus.cc:2454] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:58.305312 27446 raft_consensus.cc:2454] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:58:58.305588 24566 leader_election.cc:302] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:58.305752 1479 raft_consensus.cc:2781] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:58.305817 1479 raft_consensus.cc:482] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:58.305852 1479 raft_consensus.cc:3037] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.307180 1479 raft_consensus.cc:504] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:58.307509 1479 leader_election.cc:288] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:58.307824 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966d864897b46a584b0ba863940f731" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:58.307935 27177 raft_consensus.cc:3037] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.307916 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4966d864897b46a584b0ba863940f731" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:58.308010 27446 raft_consensus.cc:3037] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.308916 27177 raft_consensus.cc:2454] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:58.308924 27446 raft_consensus.cc:2454] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:58:58.309167 24567 leader_election.cc:302] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:58:58.309319 1479 raft_consensus.cc:2781] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:58.309396 1479 raft_consensus.cc:686] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:58:58.309547 1479 consensus_queue.cc:237] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:58.312237 24479 catalog_manager.cc:5484] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:58.339591 1530 consensus_queue.cc:1035] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.342932 1376 consensus_queue.cc:1035] T 65e9454c295a4a0cac4b955b916cd547 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.361536 1538 consensus_queue.cc:1035] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.365010 1538 consensus_queue.cc:1035] T bd8faf173a6e43d5be01ce66ff16e5c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.379173 1479 consensus_queue.cc:1035] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.382413 1479 consensus_queue.cc:1035] T d0c09a31b1ba43639e1290bca14eefad P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.428790 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:58.433225 27446 raft_consensus.cc:1261] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:58.433324 27177 raft_consensus.cc:1261] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:58.433959 1479 consensus_queue.cc:1035] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.434286 1601 consensus_queue.cc:1035] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:58.437625 24483 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]). Expected version 1 got 0
I20240502 01:58:58.438041 24479 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]). Expected version 1 got 0
I20240502 01:58:58.440230 1597 mvcc.cc:205] Tried to move back new op lower bound from 7023063607016845312 to 7023063606517055488. Current Snapshot: MvccSnapshot[applied={T|T < 7023063607016845312}]
I20240502 01:58:58.441246 24483 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]). Expected version 1 got 0
I20240502 01:58:58.443148 24483 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]). Expected version 1 got 0
I20240502 01:58:58.443127 1607 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:58.443421 1607 tablet.cc:1620] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:58.445842 1608 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:58.446125 1608 tablet.cc:1620] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:58.446112 24483 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc]). Expected version 1 got 0
I20240502 01:58:58.445066 1609 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:58.447170 1609 tablet.cc:1620] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:58.448525 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc] alter complete (version 1)
W20240502 01:58:58.465378 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet 252502ad2e124996b345888706e674d5: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:58.483675 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:58.489413 25010 tablet_service.cc:1467] Processing CreateTablet for tablet c5c4432030444820be813dd1c4a203ad (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.489696 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c4432030444820be813dd1c4a203ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.491879 1612 tablet_bootstrap.cc:492] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:58.492187 24743 tablet_service.cc:1467] Processing CreateTablet for tablet c5c4432030444820be813dd1c4a203ad (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.492445 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c4432030444820be813dd1c4a203ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.492837 1612 tablet_bootstrap.cc:654] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.493818 1612 tablet_bootstrap.cc:492] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:58.493901 1612 ts_tablet_manager.cc:1367] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:58.494377 1612 raft_consensus.cc:348] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.494484 1612 raft_consensus.cc:374] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.494518 1612 raft_consensus.cc:729] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.494621 1612 consensus_queue.cc:260] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.494741 27558 tablet_service.cc:1467] Processing CreateTablet for tablet c5c4432030444820be813dd1c4a203ad (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.494904 1612 ts_tablet_manager.cc:1398] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:58.495018 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c4432030444820be813dd1c4a203ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.495316 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.495527 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.495548 24875 tablet_service.cc:1467] Processing CreateTablet for tablet 82e76518139b4e148254815678b16973 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.495782 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e76518139b4e148254815678b16973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.495784 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.496042 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.498075 1616 tablet_bootstrap.cc:492] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:58.498723 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 82e76518139b4e148254815678b16973 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.498982 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e76518139b4e148254815678b16973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.499202 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 82e76518139b4e148254815678b16973 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.499455 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e76518139b4e148254815678b16973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.499648 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=5f0c26ffae3b45648491581dae7b7060]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:58.499945 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.501291 1617 tablet_bootstrap.cc:492] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:58.502607 1617 tablet_bootstrap.cc:654] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.502740 1616 tablet_bootstrap.cc:654] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.503976 1617 tablet_bootstrap.cc:492] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:58.504110 1617 ts_tablet_manager.cc:1367] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:58.504573 1617 raft_consensus.cc:348] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.504739 1617 raft_consensus.cc:374] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.504812 1617 raft_consensus.cc:729] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.504976 1617 consensus_queue.cc:260] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.506076 1617 ts_tablet_manager.cc:1398] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:58.506489 1616 tablet_bootstrap.cc:492] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:58.506615 1616 ts_tablet_manager.cc:1367] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 01:58:58.507263 1616 raft_consensus.cc:348] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.507462 1616 raft_consensus.cc:374] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.507458 1624 tablet_bootstrap.cc:492] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:58.507540 1616 raft_consensus.cc:729] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.507748 1616 consensus_queue.cc:260] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.508209 1614 tablet_bootstrap.cc:492] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:58.509680 1614 tablet_bootstrap.cc:654] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.510416 1616 ts_tablet_manager.cc:1398] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:58.510866 1615 tablet_bootstrap.cc:492] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:58.511649 1616 tablet_bootstrap.cc:492] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:58.511965 1624 tablet_bootstrap.cc:654] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.512565 1616 tablet_bootstrap.cc:654] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.513701 1616 tablet_bootstrap.cc:492] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:58.513723 1615 tablet_bootstrap.cc:654] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.513789 1616 ts_tablet_manager.cc:1367] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:58.514261 1616 raft_consensus.cc:348] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.514408 1616 raft_consensus.cc:374] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.514496 1616 raft_consensus.cc:729] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.514621 1616 consensus_queue.cc:260] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.514706 1624 tablet_bootstrap.cc:492] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:58.514814 1624 ts_tablet_manager.cc:1367] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:58.514963 1616 ts_tablet_manager.cc:1398] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:58.515264 1624 raft_consensus.cc:348] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.515481 1624 raft_consensus.cc:374] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.515870 1615 tablet_bootstrap.cc:492] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:58.516002 1615 ts_tablet_manager.cc:1367] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:58.516503 1615 raft_consensus.cc:348] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.517006 1614 tablet_bootstrap.cc:492] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:58.517099 1614 ts_tablet_manager.cc:1367] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:58.517544 1614 raft_consensus.cc:348] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.517794 1624 raft_consensus.cc:729] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.517954 1624 consensus_queue.cc:260] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.518088 1614 raft_consensus.cc:374] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.518146 1614 raft_consensus.cc:729] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.518271 1614 consensus_queue.cc:260] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.518579 1614 ts_tablet_manager.cc:1398] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:58.518872 1624 ts_tablet_manager.cc:1398] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:58.518983 1615 raft_consensus.cc:374] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.519068 1615 raft_consensus.cc:729] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.519237 1614 tablet_bootstrap.cc:492] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:58.519296 1615 consensus_queue.cc:260] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.519665 1615 ts_tablet_manager.cc:1398] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:58.519970 1615 tablet_bootstrap.cc:492] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:58.520819 1615 tablet_bootstrap.cc:654] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.521802 1615 tablet_bootstrap.cc:492] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:58.521875 1615 ts_tablet_manager.cc:1367] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:58.522292 1615 raft_consensus.cc:348] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.522377 1615 raft_consensus.cc:374] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.522404 1615 raft_consensus.cc:729] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.522673 1615 consensus_queue.cc:260] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.522979 1615 ts_tablet_manager.cc:1398] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:58.524044 1614 tablet_bootstrap.cc:654] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.525010 1614 tablet_bootstrap.cc:492] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:58.525116 1614 ts_tablet_manager.cc:1367] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:58.525583 1614 raft_consensus.cc:348] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.525688 1614 raft_consensus.cc:374] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.525719 1614 raft_consensus.cc:729] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.525823 1614 consensus_queue.cc:260] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.526103 1614 ts_tablet_manager.cc:1398] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:58.550546 1376 raft_consensus.cc:482] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:58.550626 1376 raft_consensus.cc:504] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.551297 1376 leader_election.cc:288] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:58.551589 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e76518139b4e148254815678b16973" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:58.551581 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e76518139b4e148254815678b16973" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:58.551698 27578 raft_consensus.cc:2454] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:58.551700 27177 raft_consensus.cc:2454] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:58:58.551951 24832 leader_election.cc:302] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:58.552112 1376 raft_consensus.cc:2781] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:58.552165 1376 raft_consensus.cc:482] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:58.552201 1376 raft_consensus.cc:3037] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.552201 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:58.553151 1376 raft_consensus.cc:504] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.553483 1376 leader_election.cc:288] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:58.554069 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e76518139b4e148254815678b16973" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:58.554162 27177 raft_consensus.cc:3037] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.554294 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e76518139b4e148254815678b16973" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:58.554399 27578 raft_consensus.cc:3037] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.554996 27177 raft_consensus.cc:2454] T 82e76518139b4e148254815678b16973 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:58.555315 27578 raft_consensus.cc:2454] T 82e76518139b4e148254815678b16973 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:58:58.555552 24832 leader_election.cc:302] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:58.556295 1376 raft_consensus.cc:2781] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:58.556376 1376 raft_consensus.cc:686] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:58:58.556486 1376 consensus_queue.cc:237] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:58.559355 24479 catalog_manager.cc:5484] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:58.559870 1607 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:58.560165 1607 tablet.cc:1620] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:58:58.561882 1608 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:58.562186 1608 tablet.cc:1620] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:58:58.563972 1609 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:58.564267 1609 tablet.cc:1620] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:58.564517 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9afc2b37334243008377286c5e34decc] alter complete (version 2)
I20240502 01:58:58.564558 1278 raft_consensus.cc:482] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:58.564622 1278 raft_consensus.cc:504] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.564994 1278 leader_election.cc:288] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:58.565235 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c4432030444820be813dd1c4a203ad" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:58.565354 25032 raft_consensus.cc:2454] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:58.565335 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c4432030444820be813dd1c4a203ad" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:58.565460 24765 raft_consensus.cc:2454] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:58:58.565757 27514 leader_election.cc:302] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:58.566084 1278 raft_consensus.cc:2781] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:58.566155 1278 raft_consensus.cc:482] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:58.566191 1278 raft_consensus.cc:3037] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.566884 1278 raft_consensus.cc:504] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.567243 1278 leader_election.cc:288] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:58.567443 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c4432030444820be813dd1c4a203ad" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:58.567549 25032 raft_consensus.cc:3037] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.567624 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c4432030444820be813dd1c4a203ad" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:58.567705 24765 raft_consensus.cc:3037] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.568424 24765 raft_consensus.cc:2454] T c5c4432030444820be813dd1c4a203ad P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:58.568521 25032 raft_consensus.cc:2454] T c5c4432030444820be813dd1c4a203ad P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:58:58.568683 27514 leader_election.cc:302] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:58.568878 1278 raft_consensus.cc:2781] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:58.568975 1278 raft_consensus.cc:686] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:58:58.569103 1278 consensus_queue.cc:237] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:58:58.571012 24479 catalog_manager.cc:5484] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:58.579335 1445 raft_consensus.cc:482] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:58.579420 1445 raft_consensus.cc:504] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.579799 1445 leader_election.cc:288] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:58.580116 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2d2c4d8bc4a45b3a2d10b63c5d2dd" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:58.580214 27444 raft_consensus.cc:2454] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:58.580428 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2d2c4d8bc4a45b3a2d10b63c5d2dd" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:58.580454 24701 leader_election.cc:302] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:58.580543 24896 raft_consensus.cc:2454] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:58.580600 1445 raft_consensus.cc:2781] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:58.580657 1445 raft_consensus.cc:482] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:58.580701 1445 raft_consensus.cc:3037] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.581410 1445 raft_consensus.cc:504] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.581717 1445 leader_election.cc:288] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:58:58.582042 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2d2c4d8bc4a45b3a2d10b63c5d2dd" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:58.582042 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2d2c4d8bc4a45b3a2d10b63c5d2dd" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:58.582144 24896 raft_consensus.cc:3037] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.582144 27444 raft_consensus.cc:3037] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:58.583048 27444 raft_consensus.cc:2454] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:58.583048 24896 raft_consensus.cc:2454] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:58.583293 24701 leader_election.cc:302] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:58.583433 1445 raft_consensus.cc:2781] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:58.583578 1445 raft_consensus.cc:686] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:58.583703 1445 consensus_queue.cc:237] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:58:58.585785 24479 catalog_manager.cc:5484] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:58.588238 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:58.666785 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:58.672470 1607 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:58.672919 1608 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:58.672964 1609 tablet.cc:1722] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:58.678709 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=9afc2b37334243008377286c5e34decc] alter complete (version 3)
W20240502 01:58:58.683833 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:58.730809 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:58.782464 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:58.782634 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:58.784615 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 4966d864897b46a584b0ba863940f731
I20240502 01:58:58.785246 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet 4966d864897b46a584b0ba863940f731 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:58 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:58.785243 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4966d864897b46a584b0ba863940f731 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:58 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:58:58.785274 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 4966d864897b46a584b0ba863940f731 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:58 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:58:58.785581 1682 tablet_replica.cc:330] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:58.785717 1682 raft_consensus.cc:2229] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:58.785815 1682 raft_consensus.cc:2258] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:58.786051 1683 tablet_replica.cc:330] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:58:58.786175 1684 tablet_replica.cc:330] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:58:58.786317 1684 raft_consensus.cc:2229] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:58.786448 1684 raft_consensus.cc:2258] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:58.786546 1683 raft_consensus.cc:2229] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:58.786800 1683 raft_consensus.cc:2258] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:58.787004 1684 ts_tablet_manager.cc:1875] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:58.787055 1682 ts_tablet_manager.cc:1875] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:58.787516 1683 ts_tablet_manager.cc:1875] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:58.789436 1682 ts_tablet_manager.cc:1888] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:58.789543 1682 log.cc:1196] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/4966d864897b46a584b0ba863940f731
I20240502 01:58:58.789614 1683 ts_tablet_manager.cc:1888] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:58.789615 1684 ts_tablet_manager.cc:1888] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:58.789690 1684 log.cc:1196] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/4966d864897b46a584b0ba863940f731
I20240502 01:58:58.789713 1683 log.cc:1196] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/4966d864897b46a584b0ba863940f731
I20240502 01:58:58.789995 1684 ts_tablet_manager.cc:1909] T 4966d864897b46a584b0ba863940f731 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:58:58.790009 1683 ts_tablet_manager.cc:1909] T 4966d864897b46a584b0ba863940f731 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:58:58.790072 1682 ts_tablet_manager.cc:1909] T 4966d864897b46a584b0ba863940f731 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:58.790796 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table_new_name [id=9afc2b37334243008377286c5e34decc]) successfully deleted
I20240502 01:58:58.791421 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table_new_name [id=9afc2b37334243008377286c5e34decc]) successfully deleted
I20240502 01:58:58.792129 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 4966d864897b46a584b0ba863940f731 (table rebalancer_dynamic_table_new_name [id=9afc2b37334243008377286c5e34decc]) successfully deleted
I20240502 01:58:58.886575 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:58.892160 24743 tablet_service.cc:1467] Processing CreateTablet for tablet eb6badf732644a5baa8d55b01c6a9482 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]), partition=
I20240502 01:58:58.892164 27157 tablet_service.cc:1467] Processing CreateTablet for tablet eb6badf732644a5baa8d55b01c6a9482 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]), partition=
I20240502 01:58:58.892444 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6badf732644a5baa8d55b01c6a9482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.892442 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6badf732644a5baa8d55b01c6a9482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.894553 1617 tablet_bootstrap.cc:492] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:58.894654 1614 tablet_bootstrap.cc:492] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:58.895313 1617 tablet_bootstrap.cc:654] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.895495 1614 tablet_bootstrap.cc:654] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.896546 1617 tablet_bootstrap.cc:492] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:58.896624 1617 ts_tablet_manager.cc:1367] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:58.896826 24875 tablet_service.cc:1467] Processing CreateTablet for tablet eb6badf732644a5baa8d55b01c6a9482 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]), partition=
I20240502 01:58:58.896987 1617 raft_consensus.cc:348] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:58.897047 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6badf732644a5baa8d55b01c6a9482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:58.897064 1617 raft_consensus.cc:374] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.897084 1617 raft_consensus.cc:729] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.897183 1617 consensus_queue.cc:260] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:58.897528 1617 ts_tablet_manager.cc:1398] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:58.899133 1616 tablet_bootstrap.cc:492] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:58:58.899993 1614 tablet_bootstrap.cc:492] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:58.900085 1614 ts_tablet_manager.cc:1367] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:58:58.900094 1616 tablet_bootstrap.cc:654] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:58.900518 1614 raft_consensus.cc:348] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:58.900621 1614 raft_consensus.cc:374] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.900651 1614 raft_consensus.cc:729] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.900755 1614 consensus_queue.cc:260] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:58.901008 1614 ts_tablet_manager.cc:1398] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:58.902426 1616 tablet_bootstrap.cc:492] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:58:58.902506 1616 ts_tablet_manager.cc:1367] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:58.902896 1616 raft_consensus.cc:348] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:58.902987 1616 raft_consensus.cc:374] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:58.903015 1616 raft_consensus.cc:729] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:58:58.903136 1616 consensus_queue.cc:260] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:58.903404 1616 ts_tablet_manager.cc:1398] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:58.964079 1656 consensus_queue.cc:1035] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:58.964318 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:58.968866 1656 consensus_queue.cc:1035] T 82e76518139b4e148254815678b16973 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:58.996057 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:59.044075 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:59.071878 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:59.090411 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:59.096781 27558 tablet_service.cc:1467] Processing CreateTablet for tablet c0393a7a32f848af9149b3642918bb2c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.097141 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0393a7a32f848af9149b3642918bb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.097419 27423 tablet_service.cc:1467] Processing CreateTablet for tablet c0393a7a32f848af9149b3642918bb2c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.097723 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0393a7a32f848af9149b3642918bb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.098209 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 309c743b51cf4ed0b6915f1e2582da51 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.098501 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309c743b51cf4ed0b6915f1e2582da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.100200 1718 tablet_bootstrap.cc:492] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:59.100172 25010 tablet_service.cc:1467] Processing CreateTablet for tablet c0393a7a32f848af9149b3642918bb2c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.100430 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0393a7a32f848af9149b3642918bb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.100823 1719 tablet_bootstrap.cc:492] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:59.101774 1718 tablet_bootstrap.cc:654] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.103399 1718 tablet_bootstrap.cc:492] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:59.103394 27558 tablet_service.cc:1467] Processing CreateTablet for tablet f58c9512c96d4ba0addb329389592237 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.103564 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 309c743b51cf4ed0b6915f1e2582da51 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.103641 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58c9512c96d4ba0addb329389592237. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.103822 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309c743b51cf4ed0b6915f1e2582da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.103776 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 309c743b51cf4ed0b6915f1e2582da51 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.104025 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309c743b51cf4ed0b6915f1e2582da51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.104784 1718 ts_tablet_manager.cc:1367] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:59.105329 1718 raft_consensus.cc:348] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.105548 1718 raft_consensus.cc:374] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.105595 1718 raft_consensus.cc:729] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.105638 1719 tablet_bootstrap.cc:654] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.105710 1718 consensus_queue.cc:260] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.106171 27157 tablet_service.cc:1467] Processing CreateTablet for tablet f58c9512c96d4ba0addb329389592237 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.106240 1718 ts_tablet_manager.cc:1398] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:59.106357 1721 tablet_bootstrap.cc:492] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:59.106391 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58c9512c96d4ba0addb329389592237. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.107044 1617 tablet_bootstrap.cc:492] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:59.107283 1721 tablet_bootstrap.cc:654] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.108148 1614 tablet_bootstrap.cc:492] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:58:59.108296 1617 tablet_bootstrap.cc:654] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.108528 1721 tablet_bootstrap.cc:492] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:59.108604 1721 ts_tablet_manager.cc:1367] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:59.109252 27423 tablet_service.cc:1467] Processing CreateTablet for tablet f58c9512c96d4ba0addb329389592237 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2e5286149e6a48f98080469e8b03dbfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:59.109314 1614 tablet_bootstrap.cc:654] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.109336 1721 raft_consensus.cc:348] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.109431 1721 raft_consensus.cc:374] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.109460 1721 raft_consensus.cc:729] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.109498 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58c9512c96d4ba0addb329389592237. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.109673 1721 consensus_queue.cc:260] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.110872 1721 ts_tablet_manager.cc:1398] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:59.112063 1718 tablet_bootstrap.cc:492] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:58:59.112112 1617 tablet_bootstrap.cc:492] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:59.112195 1617 ts_tablet_manager.cc:1367] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:59.112617 1617 raft_consensus.cc:348] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.112709 1617 raft_consensus.cc:374] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.112738 1617 raft_consensus.cc:729] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.112844 1617 consensus_queue.cc:260] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.113087 1718 tablet_bootstrap.cc:654] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.113121 1617 ts_tablet_manager.cc:1398] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:59.114265 1718 tablet_bootstrap.cc:492] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:58:59.114346 1718 ts_tablet_manager.cc:1367] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:59.114616 1617 tablet_bootstrap.cc:492] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:59.114845 1718 raft_consensus.cc:348] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.114949 1718 raft_consensus.cc:374] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.114985 1718 raft_consensus.cc:729] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.115104 1718 consensus_queue.cc:260] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.115418 1718 ts_tablet_manager.cc:1398] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:59.115532 1617 tablet_bootstrap.cc:654] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.115733 1614 tablet_bootstrap.cc:492] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:58:59.115823 1614 ts_tablet_manager.cc:1367] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:58:59.116377 1614 raft_consensus.cc:348] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.116519 1614 raft_consensus.cc:374] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.116552 1614 raft_consensus.cc:729] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.116678 1614 consensus_queue.cc:260] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.116950 1614 ts_tablet_manager.cc:1398] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:59.117960 1717 tablet_bootstrap.cc:492] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:59.119935 1717 tablet_bootstrap.cc:654] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.121071 1719 tablet_bootstrap.cc:492] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:59.121096 1717 tablet_bootstrap.cc:492] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:59.121168 1719 ts_tablet_manager.cc:1367] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 01:58:59.121173 1717 ts_tablet_manager.cc:1367] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:59.121634 1719 raft_consensus.cc:348] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.121791 1719 raft_consensus.cc:374] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.121851 1719 raft_consensus.cc:729] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.122025 1719 consensus_queue.cc:260] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.121793 1717 raft_consensus.cc:348] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.122298 1717 raft_consensus.cc:374] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.122382 1717 raft_consensus.cc:729] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.122581 1717 consensus_queue.cc:260] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.122943 1717 ts_tablet_manager.cc:1398] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:59.123107 1719 ts_tablet_manager.cc:1398] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:59.123150 1717 tablet_bootstrap.cc:492] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:58:59.124059 1717 tablet_bootstrap.cc:654] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.125293 1278 consensus_queue.cc:1035] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.127836 1617 tablet_bootstrap.cc:492] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:59.127915 1617 ts_tablet_manager.cc:1367] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240502 01:58:59.128371 1617 raft_consensus.cc:348] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.128461 1617 raft_consensus.cc:374] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.128489 1617 raft_consensus.cc:729] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.128602 1617 consensus_queue.cc:260] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.128871 1617 ts_tablet_manager.cc:1398] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:59.129601 1717 tablet_bootstrap.cc:492] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:58:59.129731 1717 ts_tablet_manager.cc:1367] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:58:59.130252 1717 raft_consensus.cc:348] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.130405 1717 raft_consensus.cc:374] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.130496 1717 raft_consensus.cc:729] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.130649 1717 consensus_queue.cc:260] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.130966 1717 ts_tablet_manager.cc:1398] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:59.133409 760 consensus_queue.cc:1035] T c5c4432030444820be813dd1c4a203ad P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.159914 1736 consensus_queue.cc:1035] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.163043 1654 consensus_queue.cc:1035] T 0ca2d2c4d8bc4a45b3a2d10b63c5d2dd P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:59.181288 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:59.207664 1139 raft_consensus.cc:482] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:59.207784 1139 raft_consensus.cc:504] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:59.208290 1139 leader_election.cc:288] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:59.208705 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6badf732644a5baa8d55b01c6a9482" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:58:59.208729 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6badf732644a5baa8d55b01c6a9482" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:58:59.208838 24763 raft_consensus.cc:2454] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:59.208875 24896 raft_consensus.cc:2454] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:59.209154 27113 leader_election.cc:302] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:59.209327 1139 raft_consensus.cc:2781] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:59.209389 1139 raft_consensus.cc:482] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:59.209437 1139 raft_consensus.cc:3037] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.209724 851 raft_consensus.cc:482] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:59.209800 851 raft_consensus.cc:504] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.210204 851 leader_election.cc:288] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:59.210556 1139 raft_consensus.cc:504] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:59.210872 1139 leader_election.cc:288] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:58:59.211269 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6badf732644a5baa8d55b01c6a9482" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:58:59.211385 24896 raft_consensus.cc:3037] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.211441 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58c9512c96d4ba0addb329389592237" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:58:59.211570 27444 raft_consensus.cc:2454] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:59.211890 27113 leader_election.cc:302] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:59.211877 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6badf732644a5baa8d55b01c6a9482" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:58:59.211992 24763 raft_consensus.cc:3037] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.212046 1139 raft_consensus.cc:2781] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:59.212087 1139 raft_consensus.cc:482] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:59.212118 1139 raft_consensus.cc:3037] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.212473 24896 raft_consensus.cc:2454] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:59.212774 24763 raft_consensus.cc:2454] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:59.212803 27113 leader_election.cc:302] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:58:59.212956 851 raft_consensus.cc:2781] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:59.212996 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58c9512c96d4ba0addb329389592237" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:59.213040 1139 raft_consensus.cc:504] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.213101 851 raft_consensus.cc:686] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:59.213146 27578 raft_consensus.cc:2454] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:58:59.213259 851 consensus_queue.cc:237] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:58:59.213361 1139 leader_election.cc:288] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:58:59.214993 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58c9512c96d4ba0addb329389592237" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:59.215003 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58c9512c96d4ba0addb329389592237" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:58:59.215113 27578 raft_consensus.cc:3037] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.215113 27446 raft_consensus.cc:3037] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.216117 27578 raft_consensus.cc:2454] T f58c9512c96d4ba0addb329389592237 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:59.216114 27446 raft_consensus.cc:2454] T f58c9512c96d4ba0addb329389592237 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:58:59.216455 27113 leader_election.cc:302] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:59.216890 1745 raft_consensus.cc:2781] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:59.217275 1745 raft_consensus.cc:686] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:58:59.217479 1745 consensus_queue.cc:237] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:58:59.218190 24483 catalog_manager.cc:5484] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:59.222702 24483 catalog_manager.cc:5484] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:59.256146 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:59.314707 32105 raft_consensus.cc:482] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:59.314817 32105 raft_consensus.cc:504] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.315229 32105 leader_election.cc:288] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:59.315555 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0393a7a32f848af9149b3642918bb2c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:58:59.315603 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0393a7a32f848af9149b3642918bb2c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:58:59.315682 27578 raft_consensus.cc:2454] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:59.315752 25032 raft_consensus.cc:2454] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:58:59.316020 27377 leader_election.cc:302] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:58:59.316203 32105 raft_consensus.cc:2781] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:59.316270 32105 raft_consensus.cc:482] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:59.316316 32105 raft_consensus.cc:3037] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.317095 32105 raft_consensus.cc:504] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.317417 32105 leader_election.cc:288] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:58:59.317778 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0393a7a32f848af9149b3642918bb2c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:58:59.317790 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0393a7a32f848af9149b3642918bb2c" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:58:59.317883 25032 raft_consensus.cc:3037] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.317898 27578 raft_consensus.cc:3037] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.318764 25032 raft_consensus.cc:2454] T c0393a7a32f848af9149b3642918bb2c P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:59.318765 27578 raft_consensus.cc:2454] T c0393a7a32f848af9149b3642918bb2c P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:58:59.319016 27380 leader_election.cc:302] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:58:59.319180 32105 raft_consensus.cc:2781] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:59.319281 32105 raft_consensus.cc:686] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:58:59.319434 32105 consensus_queue.cc:237] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.321717 24483 catalog_manager.cc:5484] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:59.323093 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:59.364775 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:58:59.369280 24896 raft_consensus.cc:1261] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:59.369666 24763 raft_consensus.cc:1261] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:59.369724 1745 consensus_queue.cc:1035] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.370404 1745 consensus_queue.cc:1035] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.373406 1689 mvcc.cc:205] Tried to move back new op lower bound from 7023063610852876288 to 7023063610218729472. Current Snapshot: MvccSnapshot[applied={T|T < 7023063610852876288}]
I20240502 01:58:59.373873 24483 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]). Expected version 1 got 0
I20240502 01:58:59.373885 1771 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:59.374337 1771 tablet.cc:1620] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:59.374634 24479 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]). Expected version 1 got 0
I20240502 01:58:59.375509 1773 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:59.375811 1773 tablet.cc:1620] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:59.376135 24479 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]). Expected version 1 got 0
I20240502 01:58:59.376765 24483 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]). Expected version 1 got 0
I20240502 01:58:59.376832 1772 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:58:59.377125 1772 tablet.cc:1620] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:59.377761 24482 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12]). Expected version 1 got 0
I20240502 01:58:59.381889 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12] alter complete (version 1)
W20240502 01:58:59.406996 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:59.417841 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:59.473961 1736 raft_consensus.cc:482] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:59.474088 1736 raft_consensus.cc:504] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.474532 1736 leader_election.cc:288] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:59.474838 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309c743b51cf4ed0b6915f1e2582da51" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:58:59.474896 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309c743b51cf4ed0b6915f1e2582da51" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:58:59.474978 24632 raft_consensus.cc:2454] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:59.475028 27176 raft_consensus.cc:2454] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:58:59.475396 24700 leader_election.cc:302] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:58:59.475790 1736 raft_consensus.cc:2781] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:59.475860 1736 raft_consensus.cc:482] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:59.475910 1736 raft_consensus.cc:3037] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.477093 1736 raft_consensus.cc:504] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.477481 1736 leader_election.cc:288] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:58:59.477830 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309c743b51cf4ed0b6915f1e2582da51" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:58:59.477981 24632 raft_consensus.cc:3037] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.478117 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309c743b51cf4ed0b6915f1e2582da51" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:58:59.478228 27176 raft_consensus.cc:3037] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:59.479032 24632 raft_consensus.cc:2454] T 309c743b51cf4ed0b6915f1e2582da51 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:59.479190 27176 raft_consensus.cc:2454] T 309c743b51cf4ed0b6915f1e2582da51 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:58:59.480122 24700 leader_election.cc:302] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:58:59.480413 1736 raft_consensus.cc:2781] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:59.480569 1736 raft_consensus.cc:686] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:58:59.480710 1736 consensus_queue.cc:237] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:58:59.482827 24482 catalog_manager.cc:5484] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:59.485862 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:58:59.491330 1771 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:59.491643 1771 tablet.cc:1620] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:58:59.492978 1773 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:59.493319 1773 tablet.cc:1620] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:58:59.495241 1772 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:58:59.495515 1772 tablet.cc:1620] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:58:59.499567 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=9bcf67e095384eae9f74e64061eaec12] alter complete (version 2)
W20240502 01:58:59.541204 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:59.547380 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:59.600811 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:58:59.606333 1771 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:59.607340 1772 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:59.610337 1773 tablet.cc:1722] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:58:59.613062 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=9bcf67e095384eae9f74e64061eaec12] alter complete (version 3)
W20240502 01:58:59.658612 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:59.668726 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:58:59.717133 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:59.717272 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:58:59.719419 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet eb6badf732644a5baa8d55b01c6a9482
I20240502 01:58:59.720144 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6badf732644a5baa8d55b01c6a9482 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:59 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:58:59.720239 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6badf732644a5baa8d55b01c6a9482 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:59 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:58:59.720320 24875 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6badf732644a5baa8d55b01c6a9482 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:58:59 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:58:59.720547 1830 tablet_replica.cc:330] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:58:59.720613 1831 tablet_replica.cc:330] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:58:59.720650 1829 tablet_replica.cc:330] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:58:59.720698 1830 raft_consensus.cc:2229] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:59.720773 1831 raft_consensus.cc:2229] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:59.720809 1829 raft_consensus.cc:2229] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:58:59.720849 1830 raft_consensus.cc:2258] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:59.720907 1831 raft_consensus.cc:2258] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:59.721052 1829 raft_consensus.cc:2258] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:59.721526 1831 ts_tablet_manager.cc:1875] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:59.721598 1829 ts_tablet_manager.cc:1875] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:59.722303 1830 ts_tablet_manager.cc:1875] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:58:59.723697 1829 ts_tablet_manager.cc:1888] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:59.723699 1831 ts_tablet_manager.cc:1888] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:59.723769 1831 log.cc:1196] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb6badf732644a5baa8d55b01c6a9482
I20240502 01:58:59.723773 1829 log.cc:1196] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb6badf732644a5baa8d55b01c6a9482
I20240502 01:58:59.724059 1829 ts_tablet_manager.cc:1909] T eb6badf732644a5baa8d55b01c6a9482 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:58:59.724128 1831 ts_tablet_manager.cc:1909] T eb6badf732644a5baa8d55b01c6a9482 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:58:59.724484 1830 ts_tablet_manager.cc:1888] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:58:59.724555 1830 log.cc:1196] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb6badf732644a5baa8d55b01c6a9482
I20240502 01:58:59.724737 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table_new_name [id=9bcf67e095384eae9f74e64061eaec12]) successfully deleted
I20240502 01:58:59.724867 1830 ts_tablet_manager.cc:1909] T eb6badf732644a5baa8d55b01c6a9482 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:58:59.725029 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table_new_name [id=9bcf67e095384eae9f74e64061eaec12]) successfully deleted
I20240502 01:58:59.725484 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet eb6badf732644a5baa8d55b01c6a9482 (table rebalancer_dynamic_table_new_name [id=9bcf67e095384eae9f74e64061eaec12]) successfully deleted
I20240502 01:58:59.736649 851 consensus_queue.cc:1035] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.740038 1139 consensus_queue.cc:1035] T f58c9512c96d4ba0addb329389592237 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:59.765812 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:59.769748 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:59.783353 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:59.821381 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:58:59.826536 27157 tablet_service.cc:1467] Processing CreateTablet for tablet af7c708b3e61490e9b46ce09f0d76510 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]), partition=
I20240502 01:58:59.826615 24610 tablet_service.cc:1467] Processing CreateTablet for tablet af7c708b3e61490e9b46ce09f0d76510 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]), partition=
I20240502 01:58:59.826690 25010 tablet_service.cc:1467] Processing CreateTablet for tablet af7c708b3e61490e9b46ce09f0d76510 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]), partition=
I20240502 01:58:59.826822 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7c708b3e61490e9b46ce09f0d76510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.826862 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7c708b3e61490e9b46ce09f0d76510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.826902 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7c708b3e61490e9b46ce09f0d76510. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:59.828708 1836 tablet_bootstrap.cc:492] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:58:59.829497 1836 tablet_bootstrap.cc:654] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.829493 1838 tablet_bootstrap.cc:492] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:58:59.830332 1838 tablet_bootstrap.cc:654] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.830440 1836 tablet_bootstrap.cc:492] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:58:59.830513 1836 ts_tablet_manager.cc:1367] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:59.830878 1836 raft_consensus.cc:348] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.830940 1839 tablet_bootstrap.cc:492] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:58:59.831159 1836 raft_consensus.cc:374] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.831219 1836 raft_consensus.cc:729] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.831660 1836 consensus_queue.cc:260] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.831748 1839 tablet_bootstrap.cc:654] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:59.831943 1836 ts_tablet_manager.cc:1398] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:59.832616 1838 tablet_bootstrap.cc:492] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:58:59.832736 1838 ts_tablet_manager.cc:1367] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:59.832763 1839 tablet_bootstrap.cc:492] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:58:59.832855 1839 ts_tablet_manager.cc:1367] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:59.833181 1838 raft_consensus.cc:348] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.833276 1838 raft_consensus.cc:374] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.833330 1838 raft_consensus.cc:729] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.833329 1839 raft_consensus.cc:348] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.833487 1839 raft_consensus.cc:374] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:59.833454 1838 consensus_queue.cc:260] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.833529 1839 raft_consensus.cc:729] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:58:59.833781 1839 consensus_queue.cc:260] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:58:59.834297 1839 ts_tablet_manager.cc:1398] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:59.835140 991 consensus_queue.cc:1035] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.837899 991 consensus_queue.cc:1035] T c0393a7a32f848af9149b3642918bb2c P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.836877 1838 ts_tablet_manager.cc:1398] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:59.902773 1736 consensus_queue.cc:1035] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:59.908948 1738 consensus_queue.cc:1035] T 309c743b51cf4ed0b6915f1e2582da51 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:00.008992 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:59:00.021260 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:59:00.021878 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.024585 1139 raft_consensus.cc:482] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:00.024689 1139 raft_consensus.cc:504] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.025059 1139 leader_election.cc:288] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:00.025381 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7c708b3e61490e9b46ce09f0d76510" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:00.025383 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7c708b3e61490e9b46ce09f0d76510" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:00.025538 25032 raft_consensus.cc:2454] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:00.025552 24632 raft_consensus.cc:2454] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:00.025889 27110 leader_election.cc:302] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:00.026086 1139 raft_consensus.cc:2781] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:00.026134 1139 raft_consensus.cc:482] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:00.026189 1139 raft_consensus.cc:3037] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.027081 1139 raft_consensus.cc:504] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.027374 1139 leader_election.cc:288] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:00.027720 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7c708b3e61490e9b46ce09f0d76510" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:00.027849 25032 raft_consensus.cc:3037] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.027819 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7c708b3e61490e9b46ce09f0d76510" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:00.027942 24632 raft_consensus.cc:3037] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:59:00.028195 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.028944 24632 raft_consensus.cc:2454] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:00.028944 25032 raft_consensus.cc:2454] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:00.029224 27110 leader_election.cc:302] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:00.029436 1139 raft_consensus.cc:2781] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:00.029559 1139 raft_consensus.cc:686] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:59:00.029891 1139 consensus_queue.cc:237] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
W20240502 01:59:00.030246 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 01:59:00.032548 24482 catalog_manager.cc:5484] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:00.037214 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.040603 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 01:59:00.041234 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet da60bdbf6831421e9f9abec53c6e1d35: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:00.064872 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:59:00.067299 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:00.069972 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 01:59:00.071432 24743 tablet_service.cc:1467] Processing CreateTablet for tablet c1c0477d0de4498f8a8e08e3845a9123 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.071534 24875 tablet_service.cc:1467] Processing CreateTablet for tablet c1c0477d0de4498f8a8e08e3845a9123 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.071746 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c0477d0de4498f8a8e08e3845a9123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.071743 27558 tablet_service.cc:1467] Processing CreateTablet for tablet c1c0477d0de4498f8a8e08e3845a9123 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.071808 24875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c0477d0de4498f8a8e08e3845a9123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.071974 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c0477d0de4498f8a8e08e3845a9123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.072609 24877 tablet_service.cc:1467] Processing CreateTablet for tablet c10a10bbcc4b4d878a57f063127dd01b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.072825 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10a10bbcc4b4d878a57f063127dd01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.074352 27423 tablet_service.cc:1467] Processing CreateTablet for tablet c10a10bbcc4b4d878a57f063127dd01b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.074370 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc3ae7ea53c4dc6a58f1f1e7618f817 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.074589 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10a10bbcc4b4d878a57f063127dd01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.074625 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc3ae7ea53c4dc6a58f1f1e7618f817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.075209 1879 tablet_bootstrap.cc:492] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:00.076125 1879 tablet_bootstrap.cc:654] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.076303 1881 tablet_bootstrap.cc:492] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:00.076953 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc3ae7ea53c4dc6a58f1f1e7618f817 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.077085 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc3ae7ea53c4dc6a58f1f1e7618f817 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.077234 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc3ae7ea53c4dc6a58f1f1e7618f817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.077342 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc3ae7ea53c4dc6a58f1f1e7618f817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.077327 25010 tablet_service.cc:1467] Processing CreateTablet for tablet c10a10bbcc4b4d878a57f063127dd01b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=13f3b5e75e0a4b5594ffc2c65cf7bdfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.077538 1880 tablet_bootstrap.cc:492] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:00.077600 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10a10bbcc4b4d878a57f063127dd01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.078570 1880 tablet_bootstrap.cc:654] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.079526 1836 tablet_bootstrap.cc:492] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:00.079684 1838 tablet_bootstrap.cc:492] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:00.080056 1878 tablet_bootstrap.cc:492] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:00.080437 1836 tablet_bootstrap.cc:654] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.080547 1881 tablet_bootstrap.cc:654] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.080655 1838 tablet_bootstrap.cc:654] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.081059 1878 tablet_bootstrap.cc:654] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.081177 1839 tablet_bootstrap.cc:492] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:00.082005 1839 tablet_bootstrap.cc:654] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.082171 1878 tablet_bootstrap.cc:492] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:00.082266 1878 ts_tablet_manager.cc:1367] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:00.082731 1878 raft_consensus.cc:348] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.082852 1878 raft_consensus.cc:374] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.082928 1878 raft_consensus.cc:729] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.083148 1878 consensus_queue.cc:260] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.083305 1838 tablet_bootstrap.cc:492] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:00.083387 1838 ts_tablet_manager.cc:1367] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:59:00.083423 1880 tablet_bootstrap.cc:492] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:00.083514 1880 ts_tablet_manager.cc:1367] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:00.083536 1878 ts_tablet_manager.cc:1398] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.083823 1838 raft_consensus.cc:348] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.083920 1838 raft_consensus.cc:374] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.083948 1838 raft_consensus.cc:729] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.083961 1836 tablet_bootstrap.cc:492] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:00.084059 1836 ts_tablet_manager.cc:1367] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:00.084065 1838 consensus_queue.cc:260] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.084367 1838 ts_tablet_manager.cc:1398] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:00.084483 1839 tablet_bootstrap.cc:492] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:00.084484 1836 raft_consensus.cc:348] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.084568 1839 ts_tablet_manager.cc:1367] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:59:00.084589 1836 raft_consensus.cc:374] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.084621 1836 raft_consensus.cc:729] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.084729 1836 consensus_queue.cc:260] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.085013 1836 ts_tablet_manager.cc:1398] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.085158 1839 raft_consensus.cc:348] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.085253 1839 raft_consensus.cc:374] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.085280 1839 raft_consensus.cc:729] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.085386 1839 consensus_queue.cc:260] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.085655 1839 ts_tablet_manager.cc:1398] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:00.087285 1880 raft_consensus.cc:348] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.087435 1880 raft_consensus.cc:374] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.087502 1880 raft_consensus.cc:729] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.087708 1880 consensus_queue.cc:260] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.088099 1880 ts_tablet_manager.cc:1398] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:00.089058 1881 tablet_bootstrap.cc:492] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:00.089141 1881 ts_tablet_manager.cc:1367] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:59:00.089594 1881 raft_consensus.cc:348] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.089705 1881 raft_consensus.cc:374] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.089738 1881 raft_consensus.cc:729] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.089848 1881 consensus_queue.cc:260] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.090262 1881 ts_tablet_manager.cc:1398] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:00.091542 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.097627 1879 tablet_bootstrap.cc:492] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:00.097720 1879 ts_tablet_manager.cc:1367] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 01:59:00.098160 1879 raft_consensus.cc:348] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.098255 1879 raft_consensus.cc:374] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.098275 1879 raft_consensus.cc:729] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.098430 1879 consensus_queue.cc:260] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.098716 1879 ts_tablet_manager.cc:1398] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.098975 1879 tablet_bootstrap.cc:492] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:00.099932 1879 tablet_bootstrap.cc:654] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.101052 1879 tablet_bootstrap.cc:492] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:00.101121 1879 ts_tablet_manager.cc:1367] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:00.101567 1879 raft_consensus.cc:348] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.101671 1879 raft_consensus.cc:374] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.101750 1879 raft_consensus.cc:729] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.101967 1879 consensus_queue.cc:260] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.102342 1879 ts_tablet_manager.cc:1398] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.102502 1879 tablet_bootstrap.cc:492] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:00.103222 1879 tablet_bootstrap.cc:654] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.104473 1879 tablet_bootstrap.cc:492] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:00.104560 1879 ts_tablet_manager.cc:1367] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:00.104967 1879 raft_consensus.cc:348] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.105208 1879 raft_consensus.cc:374] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.105299 1879 raft_consensus.cc:729] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
W20240502 01:59:00.105409 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 01:59:00.105489 1879 consensus_queue.cc:260] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.105803 1879 ts_tablet_manager.cc:1398] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:00.106137 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.111248 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.117712 1800 raft_consensus.cc:482] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:00.117810 1800 raft_consensus.cc:504] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.118343 1800 leader_election.cc:288] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:00.118641 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a10bbcc4b4d878a57f063127dd01b" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:00.118690 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a10bbcc4b4d878a57f063127dd01b" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:00.118775 27446 raft_consensus.cc:2454] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:00.118815 25032 raft_consensus.cc:2454] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:00.119043 24833 leader_election.cc:302] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:00.119202 1800 raft_consensus.cc:2781] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:00.119256 1800 raft_consensus.cc:482] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:00.119297 1800 raft_consensus.cc:3037] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.120205 1800 raft_consensus.cc:504] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.120478 1800 leader_election.cc:288] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:00.120788 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a10bbcc4b4d878a57f063127dd01b" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:00.120885 25032 raft_consensus.cc:3037] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.121573 25032 raft_consensus.cc:2454] T c10a10bbcc4b4d878a57f063127dd01b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:00.121774 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a10bbcc4b4d878a57f063127dd01b" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:00.121838 24830 leader_election.cc:302] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:00.121891 27446 raft_consensus.cc:3037] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.122040 1800 raft_consensus.cc:2781] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:00.122130 1800 raft_consensus.cc:686] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:00.122248 1800 consensus_queue.cc:237] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:00.122754 27446 raft_consensus.cc:2454] T c10a10bbcc4b4d878a57f063127dd01b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:00.124217 24482 catalog_manager.cc:5484] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:00.127655 1280 raft_consensus.cc:482] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:00.127760 1280 raft_consensus.cc:504] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.128198 1280 leader_election.cc:288] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:00.128481 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc3ae7ea53c4dc6a58f1f1e7618f817" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:00.128607 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc3ae7ea53c4dc6a58f1f1e7618f817" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:00.128605 27176 raft_consensus.cc:2454] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:00.128710 24897 raft_consensus.cc:2454] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:00.128911 24566 leader_election.cc:302] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:00.129156 1280 raft_consensus.cc:2781] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:00.129215 1280 raft_consensus.cc:482] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:00.129256 1280 raft_consensus.cc:3037] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.130316 1280 raft_consensus.cc:504] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.130584 1280 leader_election.cc:288] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:00.130882 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc3ae7ea53c4dc6a58f1f1e7618f817" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:00.131003 27176 raft_consensus.cc:3037] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.131737 27176 raft_consensus.cc:2454] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:00.132006 24566 leader_election.cc:302] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:00.132236 1280 raft_consensus.cc:2781] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:00.132390 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc3ae7ea53c4dc6a58f1f1e7618f817" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:00.132448 1280 raft_consensus.cc:686] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:00.132491 24897 raft_consensus.cc:3037] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.132601 1280 consensus_queue.cc:237] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:00.133337 24897 raft_consensus.cc:2454] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:00.135475 24482 catalog_manager.cc:5484] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:00.154390 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 01:59:00.165197 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:00.169175 25032 raft_consensus.cc:1261] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:00.169293 24632 raft_consensus.cc:1261] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:00.169625 1139 consensus_queue.cc:1035] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.169780 851 consensus_queue.cc:1035] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.172633 1604 mvcc.cc:205] Tried to move back new op lower bound from 7023063614129434624 to 7023063613564342272. Current Snapshot: MvccSnapshot[applied={T|T < 7023063614129434624}]
I20240502 01:59:00.173605 24483 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]). Expected version 1 got 0
I20240502 01:59:00.174427 1924 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:00.174799 24482 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]). Expected version 1 got 0
I20240502 01:59:00.174927 1924 tablet.cc:1620] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:00.174906 1923 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:00.175315 24487 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]). Expected version 1 got 0
I20240502 01:59:00.175319 1923 tablet.cc:1620] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:00.177501 24483 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e]). Expected version 1 got 0
I20240502 01:59:00.180510 1925 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:00.180796 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e] alter complete (version 1)
I20240502 01:59:00.180855 1925 tablet.cc:1620] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
W20240502 01:59:00.202625 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.214860 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.222630 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.266484 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:00.283046 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:00.289213 1923 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:00.289677 1923 tablet.cc:1620] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:00.290575 1924 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:00.290872 1924 tablet.cc:1620] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:00.294456 1925 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:00.294852 1925 tablet.cc:1620] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:00.295930 24487 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=cce9b6e891614ff28a707a44f584247e] alter complete (version 2)
W20240502 01:59:00.307731 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.309351 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:00.373365 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.393765 1800 raft_consensus.cc:482] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:00.393891 1800 raft_consensus.cc:504] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.394351 1800 leader_election.cc:288] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:59:00.394716 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c0477d0de4498f8a8e08e3845a9123" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:00.394738 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c0477d0de4498f8a8e08e3845a9123" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:00.394852 24765 raft_consensus.cc:2454] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:00.394869 27578 raft_consensus.cc:2454] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:00.395145 24833 leader_election.cc:302] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:00.395293 1800 raft_consensus.cc:2781] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:00.395347 1800 raft_consensus.cc:482] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:00.395392 1800 raft_consensus.cc:3037] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.396190 1800 raft_consensus.cc:504] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.396481 1800 leader_election.cc:288] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:59:00.396842 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c0477d0de4498f8a8e08e3845a9123" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:00.396888 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c0477d0de4498f8a8e08e3845a9123" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:00.396945 27578 raft_consensus.cc:3037] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.397019 24765 raft_consensus.cc:3037] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.397797 24765 raft_consensus.cc:2454] T c1c0477d0de4498f8a8e08e3845a9123 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:00.397807 27578 raft_consensus.cc:2454] T c1c0477d0de4498f8a8e08e3845a9123 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:00.398200 24833 leader_election.cc:302] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:00.398169 24483 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:00.398514 1800 raft_consensus.cc:2781] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:00.398602 1800 raft_consensus.cc:686] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:00.398731 1800 consensus_queue.cc:237] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.401911 24482 catalog_manager.cc:5484] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:00.404450 1923 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:00.405140 1925 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:00.406898 1924 tablet.cc:1722] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:00.410987 24487 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=cce9b6e891614ff28a707a44f584247e] alter complete (version 3)
W20240502 01:59:00.430660 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:00.502048 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.513799 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:00.514014 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:00.516471 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet af7c708b3e61490e9b46ce09f0d76510
I20240502 01:59:00.517143 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet af7c708b3e61490e9b46ce09f0d76510 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:00 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:59:00.517294 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet af7c708b3e61490e9b46ce09f0d76510 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:00 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:59:00.517616 1955 tablet_replica.cc:330] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:59:00.517660 1954 tablet_replica.cc:330] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:59:00.517791 1954 raft_consensus.cc:2229] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:00.517781 1955 raft_consensus.cc:2229] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:00.517774 25010 tablet_service.cc:1514] Processing DeleteTablet for tablet af7c708b3e61490e9b46ce09f0d76510 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:00 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:00.517908 1955 raft_consensus.cc:2258] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:00.517985 1954 raft_consensus.cc:2258] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:00.518076 1956 tablet_replica.cc:330] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:00.518220 1956 raft_consensus.cc:2229] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:00.518347 1956 raft_consensus.cc:2258] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:00.518599 1955 ts_tablet_manager.cc:1875] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:00.518642 1954 ts_tablet_manager.cc:1875] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:00.518965 1956 ts_tablet_manager.cc:1875] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:00.520807 1955 ts_tablet_manager.cc:1888] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:00.520911 1955 log.cc:1196] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/af7c708b3e61490e9b46ce09f0d76510
I20240502 01:59:00.521000 1954 ts_tablet_manager.cc:1888] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:00.521080 1954 log.cc:1196] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/af7c708b3e61490e9b46ce09f0d76510
I20240502 01:59:00.521102 1956 ts_tablet_manager.cc:1888] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:00.521157 1956 log.cc:1196] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/af7c708b3e61490e9b46ce09f0d76510
I20240502 01:59:00.521235 1955 ts_tablet_manager.cc:1909] T af7c708b3e61490e9b46ce09f0d76510 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:59:00.521375 1954 ts_tablet_manager.cc:1909] T af7c708b3e61490e9b46ce09f0d76510 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:59:00.521461 1956 ts_tablet_manager.cc:1909] T af7c708b3e61490e9b46ce09f0d76510 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:00.521893 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table_new_name [id=cce9b6e891614ff28a707a44f584247e]) successfully deleted
I20240502 01:59:00.521989 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table_new_name [id=cce9b6e891614ff28a707a44f584247e]) successfully deleted
I20240502 01:59:00.522081 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet af7c708b3e61490e9b46ce09f0d76510 (table rebalancer_dynamic_table_new_name [id=cce9b6e891614ff28a707a44f584247e]) successfully deleted
W20240502 01:59:00.575493 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.618625 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:00.626159 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 3b468dcb0b594d5c85ef6f986b7c315e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]), partition=
I20240502 01:59:00.626246 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 3b468dcb0b594d5c85ef6f986b7c315e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]), partition=
I20240502 01:59:00.626420 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b468dcb0b594d5c85ef6f986b7c315e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.626530 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b468dcb0b594d5c85ef6f986b7c315e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.628536 1957 tablet_bootstrap.cc:492] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:00.628873 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 3b468dcb0b594d5c85ef6f986b7c315e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]), partition=
I20240502 01:59:00.629129 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b468dcb0b594d5c85ef6f986b7c315e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.629613 1957 tablet_bootstrap.cc:654] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.631260 1957 tablet_bootstrap.cc:492] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:00.631340 1964 tablet_bootstrap.cc:492] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:00.631345 1957 ts_tablet_manager.cc:1367] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:59:00.631788 1957 raft_consensus.cc:348] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.631975 1957 raft_consensus.cc:374] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.632036 1957 raft_consensus.cc:729] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.632256 1957 consensus_queue.cc:260] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.632620 1957 ts_tablet_manager.cc:1398] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:00.632975 1958 tablet_bootstrap.cc:492] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:00.633780 1964 tablet_bootstrap.cc:654] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.634402 1958 tablet_bootstrap.cc:654] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.635859 1958 tablet_bootstrap.cc:492] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:00.636011 1958 ts_tablet_manager.cc:1367] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:00.636456 1958 raft_consensus.cc:348] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.636565 1958 raft_consensus.cc:374] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.636606 1958 raft_consensus.cc:729] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.636613 1964 tablet_bootstrap.cc:492] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:00.636690 1964 ts_tablet_manager.cc:1367] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:59:00.636734 1958 consensus_queue.cc:260] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.637050 1958 ts_tablet_manager.cc:1398] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:00.639755 1964 raft_consensus.cc:348] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.640183 1964 raft_consensus.cc:374] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.640215 1964 raft_consensus.cc:729] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.640362 1964 consensus_queue.cc:260] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.641779 1964 ts_tablet_manager.cc:1398] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:59:00.643100 1800 consensus_queue.cc:1035] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.643218 1280 consensus_queue.cc:1035] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.649258 1280 consensus_queue.cc:1035] T 2fc3ae7ea53c4dc6a58f1f1e7618f817 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.659415 1800 consensus_queue.cc:1035] T c10a10bbcc4b4d878a57f063127dd01b P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.724579 1736 raft_consensus.cc:482] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:00.724694 1736 raft_consensus.cc:504] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.725116 1736 leader_election.cc:288] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:59:00.725452 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b468dcb0b594d5c85ef6f986b7c315e" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:00.725430 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b468dcb0b594d5c85ef6f986b7c315e" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:00.725574 27577 raft_consensus.cc:2454] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:00.725610 27444 raft_consensus.cc:2454] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:00.725867 24701 leader_election.cc:302] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:00.726056 1736 raft_consensus.cc:2781] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:00.726135 1736 raft_consensus.cc:482] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:00.726171 1736 raft_consensus.cc:3037] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.727072 1736 raft_consensus.cc:504] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.727445 1736 leader_election.cc:288] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:59:00.727703 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b468dcb0b594d5c85ef6f986b7c315e" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:00.727830 27444 raft_consensus.cc:3037] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.727811 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b468dcb0b594d5c85ef6f986b7c315e" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:00.727939 27577 raft_consensus.cc:3037] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.728871 27444 raft_consensus.cc:2454] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:00.728871 27577 raft_consensus.cc:2454] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:00.729148 24701 leader_election.cc:302] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:00.729333 1736 raft_consensus.cc:2781] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:00.729415 1736 raft_consensus.cc:686] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:59:00.729549 1736 consensus_queue.cc:237] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:00.731618 24481 catalog_manager.cc:5484] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:00.757401 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:00.838882 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:00.842607 27577 raft_consensus.cc:1261] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:00.842708 27444 raft_consensus.cc:1261] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer 92cb785c165749eca49af5784756461c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:00.842996 1736 consensus_queue.cc:1035] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.843143 1738 consensus_queue.cc:1035] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:00.846966 2002 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:00.847028 24481 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]). Expected version 1 got 0
I20240502 01:59:00.847291 2002 tablet.cc:1620] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:59:00.847718 24487 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]). Expected version 1 got 0
I20240502 01:59:00.848161 24479 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]). Expected version 1 got 0
I20240502 01:59:00.849768 2003 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:00.850080 2003 tablet.cc:1620] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:00.852118 24487 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]). Expected version 1 got 0
I20240502 01:59:00.852707 24481 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb]). Expected version 1 got 0
I20240502 01:59:00.853916 2004 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 01:59:00.854034 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:00.854239 2004 tablet.cc:1620] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:00.854274 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb] alter complete (version 1)
I20240502 01:59:00.913152 1656 consensus_queue.cc:1035] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:00.913796 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.920017 1656 consensus_queue.cc:1035] T c1c0477d0de4498f8a8e08e3845a9123 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:00.937451 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:00.942818 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:00.948598 27558 tablet_service.cc:1467] Processing CreateTablet for tablet b9b500f5522e4684bd852abf447f6afb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.948938 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b500f5522e4684bd852abf447f6afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.949165 24743 tablet_service.cc:1467] Processing CreateTablet for tablet c3aa914012c44dfc848586e79541ca40 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.949427 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aa914012c44dfc848586e79541ca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.949577 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffe081c68f54970b4adae70db386024 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.949805 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffe081c68f54970b4adae70db386024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.951442 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffe081c68f54970b4adae70db386024 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.951732 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffe081c68f54970b4adae70db386024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.952227 1957 tablet_bootstrap.cc:492] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:00.953022 1957 tablet_bootstrap.cc:654] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.953590 1964 tablet_bootstrap.cc:492] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:00.953658 1958 tablet_bootstrap.cc:492] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:00.954239 27157 tablet_service.cc:1467] Processing CreateTablet for tablet c3aa914012c44dfc848586e79541ca40 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.954368 2033 tablet_bootstrap.cc:492] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:00.954488 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aa914012c44dfc848586e79541ca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.954878 1964 tablet_bootstrap.cc:654] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.955312 2033 tablet_bootstrap.cc:654] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.955881 1964 tablet_bootstrap.cc:492] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:00.955955 1964 ts_tablet_manager.cc:1367] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:00.956354 1964 raft_consensus.cc:348] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:00.956434 1964 raft_consensus.cc:374] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.956460 1964 raft_consensus.cc:729] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.956573 2033 tablet_bootstrap.cc:492] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:00.956579 1964 consensus_queue.cc:260] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:00.956651 2033 ts_tablet_manager.cc:1367] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:00.956872 1964 ts_tablet_manager.cc:1398] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.957330 24610 tablet_service.cc:1467] Processing CreateTablet for tablet b9b500f5522e4684bd852abf447f6afb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.957597 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b500f5522e4684bd852abf447f6afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.957545 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffe081c68f54970b4adae70db386024 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.957777 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffe081c68f54970b4adae70db386024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.958300 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:00.958699 25010 tablet_service.cc:1467] Processing CreateTablet for tablet b9b500f5522e4684bd852abf447f6afb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.958959 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b500f5522e4684bd852abf447f6afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.959311 25012 tablet_service.cc:1467] Processing CreateTablet for tablet c3aa914012c44dfc848586e79541ca40 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1a91d99944e046bcad6a4983369ab8d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:00.959540 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aa914012c44dfc848586e79541ca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:00.961241 2039 tablet_bootstrap.cc:492] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:00.961256 2033 raft_consensus.cc:348] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:00.961356 2033 raft_consensus.cc:374] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.961388 2033 raft_consensus.cc:729] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.961534 2033 consensus_queue.cc:260] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:00.962134 2039 tablet_bootstrap.cc:654] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.962232 1958 tablet_bootstrap.cc:654] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.962589 2033 ts_tablet_manager.cc:1398] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:59:00.963227 1957 tablet_bootstrap.cc:492] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:00.963311 1957 ts_tablet_manager.cc:1367] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:59:00.963757 1957 raft_consensus.cc:348] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:00.963855 1957 raft_consensus.cc:374] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.963886 1957 raft_consensus.cc:729] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.963991 1957 consensus_queue.cc:260] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:00.964259 1957 ts_tablet_manager.cc:1398] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:00.964555 2037 tablet_bootstrap.cc:492] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:00.965551 2037 tablet_bootstrap.cc:654] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.967487 2002 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:00.967741 2002 tablet.cc:1620] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:59:00.968241 2038 tablet_bootstrap.cc:492] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:00.968920 1958 tablet_bootstrap.cc:492] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:00.968999 1958 ts_tablet_manager.cc:1367] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:59:00.969421 1958 raft_consensus.cc:348] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.969542 1958 raft_consensus.cc:374] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.969573 1958 raft_consensus.cc:729] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.969677 1958 consensus_queue.cc:260] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.969978 1958 ts_tablet_manager.cc:1398] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.970943 2004 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:00.971220 2004 tablet.cc:1620] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:00.972242 2003 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:00.972489 2003 tablet.cc:1620] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:00.969507 2038 tablet_bootstrap.cc:654] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.981778 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=31dc109de93e4e04940a473383dc7fdb] alter complete (version 2)
I20240502 01:59:00.982885 2037 tablet_bootstrap.cc:492] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:00.983088 2037 ts_tablet_manager.cc:1367] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240502 01:59:00.983422 2039 tablet_bootstrap.cc:492] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:00.983561 2039 ts_tablet_manager.cc:1367] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:59:00.983625 2038 tablet_bootstrap.cc:492] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:00.983615 2037 raft_consensus.cc:348] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:00.983716 2038 ts_tablet_manager.cc:1367] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 01:59:00.983728 2037 raft_consensus.cc:374] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.983760 2037 raft_consensus.cc:729] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.983870 2037 consensus_queue.cc:260] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:00.983999 2039 raft_consensus.cc:348] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.984136 2039 raft_consensus.cc:374] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.984951 2039 raft_consensus.cc:729] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.985070 2037 ts_tablet_manager.cc:1398] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:00.985142 2039 consensus_queue.cc:260] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.985519 2038 raft_consensus.cc:348] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.985677 2038 raft_consensus.cc:374] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.985754 2038 raft_consensus.cc:729] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.985963 2038 consensus_queue.cc:260] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.986301 2038 ts_tablet_manager.cc:1398] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:00.986575 2039 ts_tablet_manager.cc:1398] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:00.986763 2039 tablet_bootstrap.cc:492] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:00.987527 2038 tablet_bootstrap.cc:492] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:00.987654 2039 tablet_bootstrap.cc:654] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.988322 2038 tablet_bootstrap.cc:654] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:00.989697 2038 tablet_bootstrap.cc:492] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:00.989827 2038 ts_tablet_manager.cc:1367] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:00.990415 2038 raft_consensus.cc:348] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:00.990564 2038 raft_consensus.cc:374] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.990641 2038 raft_consensus.cc:729] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.990717 2039 tablet_bootstrap.cc:492] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:00.990801 2038 consensus_queue.cc:260] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:00.991149 2039 ts_tablet_manager.cc:1367] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:00.991612 2039 raft_consensus.cc:348] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:00.991719 2039 raft_consensus.cc:374] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:00.991780 2039 raft_consensus.cc:729] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:00.991817 2038 ts_tablet_manager.cc:1398] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:00.991901 2039 consensus_queue.cc:260] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:00.992246 2039 ts_tablet_manager.cc:1398] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:00.994208 1891 raft_consensus.cc:482] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:00.994304 1891 raft_consensus.cc:504] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.994764 1891 leader_election.cc:288] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:00.995034 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b500f5522e4684bd852abf447f6afb" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:00.995136 24630 raft_consensus.cc:2454] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:00.996091 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b500f5522e4684bd852abf447f6afb" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:00.996220 27578 raft_consensus.cc:2454] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:00.996497 24966 leader_election.cc:302] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:00.996659 1891 raft_consensus.cc:2781] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:00.996717 1891 raft_consensus.cc:482] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:00.996755 1891 raft_consensus.cc:3037] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.997743 1891 raft_consensus.cc:504] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:00.998078 1891 leader_election.cc:288] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:00.998378 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b500f5522e4684bd852abf447f6afb" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:00.998366 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b500f5522e4684bd852abf447f6afb" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:00.998488 24630 raft_consensus.cc:3037] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.998500 27578 raft_consensus.cc:3037] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:00.999176 27578 raft_consensus.cc:2454] T b9b500f5522e4684bd852abf447f6afb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:00.999241 24630 raft_consensus.cc:2454] T b9b500f5522e4684bd852abf447f6afb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:00.999420 24966 leader_election.cc:302] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:00.999616 1891 raft_consensus.cc:2781] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:00.999779 1891 raft_consensus.cc:686] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:00.999898 1891 consensus_queue.cc:237] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.001719 24481 catalog_manager.cc:5484] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:01.060293 2031 raft_consensus.cc:482] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:01.060425 2031 raft_consensus.cc:504] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.060854 2031 leader_election.cc:288] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:01.061153 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffe081c68f54970b4adae70db386024" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:01.061292 24630 raft_consensus.cc:2454] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:01.061499 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffe081c68f54970b4adae70db386024" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:01.061578 24832 leader_election.cc:302] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:01.061650 27444 raft_consensus.cc:2454] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:01.061751 2031 raft_consensus.cc:2781] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:01.061815 2031 raft_consensus.cc:482] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:01.061861 2031 raft_consensus.cc:3037] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.062911 2031 raft_consensus.cc:504] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.063201 2031 leader_election.cc:288] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:01.063465 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffe081c68f54970b4adae70db386024" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:01.063517 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffe081c68f54970b4adae70db386024" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:01.063612 27444 raft_consensus.cc:3037] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.063607 24630 raft_consensus.cc:3037] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.064528 27444 raft_consensus.cc:2454] T 1ffe081c68f54970b4adae70db386024 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:01.064528 24630 raft_consensus.cc:2454] T 1ffe081c68f54970b4adae70db386024 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:01.064766 24832 leader_election.cc:302] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:01.064918 2031 raft_consensus.cc:2781] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:01.065003 2031 raft_consensus.cc:686] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:01.065156 2031 consensus_queue.cc:237] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.067442 24481 catalog_manager.cc:5484] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:01.085494 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:01.090564 2002 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:01.091059 2004 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:01.092602 2003 tablet.cc:1722] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:01.096762 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=31dc109de93e4e04940a473383dc7fdb] alter complete (version 3)
I20240502 01:59:01.171577 2046 raft_consensus.cc:482] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:01.171710 2046 raft_consensus.cc:504] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.172188 2046 leader_election.cc:288] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:01.172494 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aa914012c44dfc848586e79541ca40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:01.172547 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aa914012c44dfc848586e79541ca40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:01.172619 27177 raft_consensus.cc:2454] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:01.172688 24763 raft_consensus.cc:2454] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:01.172891 24965 leader_election.cc:302] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:01.173123 2046 raft_consensus.cc:2781] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:01.173173 2046 raft_consensus.cc:482] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:01.173208 2046 raft_consensus.cc:3037] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.174399 2046 raft_consensus.cc:504] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.174723 2046 leader_election.cc:288] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:01.175122 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aa914012c44dfc848586e79541ca40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:01.175134 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aa914012c44dfc848586e79541ca40" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:01.175220 27177 raft_consensus.cc:3037] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.175243 24763 raft_consensus.cc:3037] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.176112 27177 raft_consensus.cc:2454] T c3aa914012c44dfc848586e79541ca40 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:01.176159 24763 raft_consensus.cc:2454] T c3aa914012c44dfc848586e79541ca40 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:01.176347 24965 leader_election.cc:302] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:01.176549 2046 raft_consensus.cc:2781] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:01.176623 2046 raft_consensus.cc:686] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:01.176749 2046 consensus_queue.cc:237] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.179003 24479 catalog_manager.cc:5484] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:01.200377 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:01.200529 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:01.202406 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 3b468dcb0b594d5c85ef6f986b7c315e
I20240502 01:59:01.203022 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b468dcb0b594d5c85ef6f986b7c315e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:01 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:59:01.203018 27558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b468dcb0b594d5c85ef6f986b7c315e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:01 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:59:01.203056 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b468dcb0b594d5c85ef6f986b7c315e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:01 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:59:01.203333 2076 tablet_replica.cc:330] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:59:01.203408 2077 tablet_replica.cc:330] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:59:01.203478 2076 raft_consensus.cc:2229] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:01.203555 2077 raft_consensus.cc:2229] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:01.203606 2076 raft_consensus.cc:2258] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.203749 2077 raft_consensus.cc:2258] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.203835 2075 tablet_replica.cc:330] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:59:01.203980 2075 raft_consensus.cc:2229] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:01.204109 2075 raft_consensus.cc:2258] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.204291 2077 ts_tablet_manager.cc:1875] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:01.204324 2076 ts_tablet_manager.cc:1875] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:01.204675 2075 ts_tablet_manager.cc:1875] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:01.206441 2075 ts_tablet_manager.cc:1888] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:01.206507 2075 log.cc:1196] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/3b468dcb0b594d5c85ef6f986b7c315e
I20240502 01:59:01.206547 2077 ts_tablet_manager.cc:1888] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:01.206627 2077 log.cc:1196] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b468dcb0b594d5c85ef6f986b7c315e
I20240502 01:59:01.206784 2075 ts_tablet_manager.cc:1909] T 3b468dcb0b594d5c85ef6f986b7c315e P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:59:01.206874 2076 ts_tablet_manager.cc:1888] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:01.206959 2076 log.cc:1196] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/3b468dcb0b594d5c85ef6f986b7c315e
I20240502 01:59:01.206983 2077 ts_tablet_manager.cc:1909] T 3b468dcb0b594d5c85ef6f986b7c315e P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:59:01.207389 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table_new_name [id=31dc109de93e4e04940a473383dc7fdb]) successfully deleted
I20240502 01:59:01.207576 2076 ts_tablet_manager.cc:1909] T 3b468dcb0b594d5c85ef6f986b7c315e P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:59:01.207733 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table_new_name [id=31dc109de93e4e04940a473383dc7fdb]) successfully deleted
I20240502 01:59:01.208247 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 3b468dcb0b594d5c85ef6f986b7c315e (table rebalancer_dynamic_table_new_name [id=31dc109de93e4e04940a473383dc7fdb]) successfully deleted
I20240502 01:59:01.304387 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:01.309510 24611 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd3df77d954a33bff826573ff79add (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]), partition=
I20240502 01:59:01.309511 24876 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd3df77d954a33bff826573ff79add (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]), partition=
I20240502 01:59:01.309593 27157 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd3df77d954a33bff826573ff79add (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]), partition=
I20240502 01:59:01.309775 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fd3df77d954a33bff826573ff79add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.309818 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fd3df77d954a33bff826573ff79add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.309775 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fd3df77d954a33bff826573ff79add. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.312116 2033 tablet_bootstrap.cc:492] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:01.312196 2038 tablet_bootstrap.cc:492] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:01.313023 2033 tablet_bootstrap.cc:654] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.313061 2038 tablet_bootstrap.cc:654] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.314404 2033 tablet_bootstrap.cc:492] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:01.314462 2037 tablet_bootstrap.cc:492] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:01.314488 2033 ts_tablet_manager.cc:1367] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:01.314930 2033 raft_consensus.cc:348] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.315045 2033 raft_consensus.cc:374] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.315079 2033 raft_consensus.cc:729] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.315183 2033 consensus_queue.cc:260] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.315338 2037 tablet_bootstrap.cc:654] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.315447 2033 ts_tablet_manager.cc:1398] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:59:01.317118 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:01.317344 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:01.318477 2037 tablet_bootstrap.cc:492] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:01.318562 2037 ts_tablet_manager.cc:1367] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:59:01.318948 2037 raft_consensus.cc:348] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.319056 2037 raft_consensus.cc:374] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.319087 2037 raft_consensus.cc:729] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.319203 2037 consensus_queue.cc:260] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.319495 2037 ts_tablet_manager.cc:1398] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:01.319981 2038 tablet_bootstrap.cc:492] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:01.320070 2038 ts_tablet_manager.cc:1367] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:59:01.320488 2038 raft_consensus.cc:348] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.320564 2038 raft_consensus.cc:374] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.320585 2038 raft_consensus.cc:729] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.320683 2038 consensus_queue.cc:260] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.320881 2038 ts_tablet_manager.cc:1398] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:01.336570 1139 raft_consensus.cc:482] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:01.336661 1139 raft_consensus.cc:504] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.337011 1139 leader_election.cc:288] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:01.337278 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd3df77d954a33bff826573ff79add" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:01.337304 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd3df77d954a33bff826573ff79add" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:01.337392 24630 raft_consensus.cc:2454] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:01.337419 24897 raft_consensus.cc:2454] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:01.337682 27113 leader_election.cc:302] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:01.337827 1139 raft_consensus.cc:2781] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:01.337882 1139 raft_consensus.cc:482] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:01.337920 1139 raft_consensus.cc:3037] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.338692 1139 raft_consensus.cc:504] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.338958 1139 leader_election.cc:288] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:01.339262 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd3df77d954a33bff826573ff79add" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:01.339363 24897 raft_consensus.cc:3037] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.339386 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd3df77d954a33bff826573ff79add" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:01.339509 24630 raft_consensus.cc:3037] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.340292 24897 raft_consensus.cc:2454] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:01.340376 24630 raft_consensus.cc:2454] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:01.340591 27113 leader_election.cc:302] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:01.340797 1139 raft_consensus.cc:2781] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:01.340910 1139 raft_consensus.cc:686] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:59:01.341039 1139 consensus_queue.cc:237] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:01.342972 24479 catalog_manager.cc:5484] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:01.402345 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35596116445411f857f52eadd963a16: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:01.441080 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:01.447698 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:01.452358 24897 raft_consensus.cc:1261] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:01.452706 24630 raft_consensus.cc:1261] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 9dd619a595e94004ac0f7f5a9054e7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:01.452746 1139 consensus_queue.cc:1035] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:01.453389 1139 consensus_queue.cc:1035] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:01.456578 24479 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]). Expected version 1 got 0
I20240502 01:59:01.456895 24481 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]). Expected version 1 got 0
I20240502 01:59:01.458119 2116 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:01.458467 2116 tablet.cc:1620] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:01.460502 2118 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:01.460681 24479 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]). Expected version 1 got 0
I20240502 01:59:01.460783 2118 tablet.cc:1620] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:01.461181 24481 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]). Expected version 1 got 0
I20240502 01:59:01.463287 24481 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51]). Expected version 1 got 0
I20240502 01:59:01.464131 2117 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:01.467976 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51] alter complete (version 1)
I20240502 01:59:01.469009 2117 tablet.cc:1620] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:01.507225 2046 consensus_queue.cc:1035] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:01.509063 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:01.511516 2121 consensus_queue.cc:1035] T b9b500f5522e4684bd852abf447f6afb P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:01.550506 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:01.570559 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:01.575021 1800 consensus_queue.cc:1035] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:01.578641 2116 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:01.578979 2116 tablet.cc:1620] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:01.582343 1800 consensus_queue.cc:1035] T 1ffe081c68f54970b4adae70db386024 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:01.584762 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1ae341243e434fd0897276debbe56e51] alter complete (version 2)
I20240502 01:59:01.588088 2118 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:01.588517 2118 tablet.cc:1620] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:01.589061 2117 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:01.589351 2117 tablet.cc:1620] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:01.608553 2121 consensus_queue.cc:1035] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:01.611265 2121 consensus_queue.cc:1035] T c3aa914012c44dfc848586e79541ca40 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:01.662233 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:01.676299 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:01.692507 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:01.697957 2116 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:01.698855 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:01.700323 2117 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:01.705077 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=1ae341243e434fd0897276debbe56e51] alter complete (version 3)
I20240502 01:59:01.706228 2118 tablet.cc:1722] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:01.707391 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7c40e2d1ad4c7d8dae6d95aad638bf (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.707682 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7c40e2d1ad4c7d8dae6d95aad638bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.708113 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7c40e2d1ad4c7d8dae6d95aad638bf (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.708180 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2cec1ee29845d394d0d64486d974a7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.708370 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7c40e2d1ad4c7d8dae6d95aad638bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.708424 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2cec1ee29845d394d0d64486d974a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.710067 2153 tablet_bootstrap.cc:492] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:01.710247 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2cec1ee29845d394d0d64486d974a7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.710536 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2cec1ee29845d394d0d64486d974a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.710539 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 046e41255bbb47b38f00573acc07f235 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.711030 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046e41255bbb47b38f00573acc07f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.711336 2153 tablet_bootstrap.cc:654] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.712527 2038 tablet_bootstrap.cc:492] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:01.713493 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7c40e2d1ad4c7d8dae6d95aad638bf (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.713721 2038 tablet_bootstrap.cc:654] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.713740 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7c40e2d1ad4c7d8dae6d95aad638bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.714372 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2cec1ee29845d394d0d64486d974a7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.714622 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2cec1ee29845d394d0d64486d974a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.714635 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 046e41255bbb47b38f00573acc07f235 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.714944 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046e41255bbb47b38f00573acc07f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.716095 2033 tablet_bootstrap.cc:492] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:01.716984 2033 tablet_bootstrap.cc:654] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.717017 2037 tablet_bootstrap.cc:492] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:01.717893 2037 tablet_bootstrap.cc:654] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.718603 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 046e41255bbb47b38f00573acc07f235 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c42b0abd522a432fa53264662d0009fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:01.718840 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046e41255bbb47b38f00573acc07f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.718856 2157 tablet_bootstrap.cc:492] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:01.719470 2154 tablet_bootstrap.cc:492] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:01.719722 2037 tablet_bootstrap.cc:492] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:01.719758 2157 tablet_bootstrap.cc:654] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.719805 2037 ts_tablet_manager.cc:1367] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:01.720230 2037 raft_consensus.cc:348] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.720322 2037 raft_consensus.cc:374] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.720342 2037 raft_consensus.cc:729] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.720453 2037 consensus_queue.cc:260] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.720497 2154 tablet_bootstrap.cc:654] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.720716 2037 ts_tablet_manager.cc:1398] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:01.721017 2158 tablet_bootstrap.cc:492] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:01.721921 2158 tablet_bootstrap.cc:654] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.721988 2153 tablet_bootstrap.cc:492] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:01.722095 2153 ts_tablet_manager.cc:1367] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:59:01.722127 2033 tablet_bootstrap.cc:492] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:01.722208 2033 ts_tablet_manager.cc:1367] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:01.722617 2038 tablet_bootstrap.cc:492] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:01.722627 2033 raft_consensus.cc:348] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.722702 2038 ts_tablet_manager.cc:1367] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:59:01.722729 2033 raft_consensus.cc:374] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.722761 2033 raft_consensus.cc:729] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.722873 2033 consensus_queue.cc:260] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.723120 2033 ts_tablet_manager.cc:1398] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:01.723178 2038 raft_consensus.cc:348] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.723230 2153 raft_consensus.cc:348] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.723299 2038 raft_consensus.cc:374] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.723325 2153 raft_consensus.cc:374] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.723332 2038 raft_consensus.cc:729] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.723460 2038 consensus_queue.cc:260] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.723356 2153 raft_consensus.cc:729] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.723752 2038 ts_tablet_manager.cc:1398] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:01.723834 2153 consensus_queue.cc:260] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.723963 2038 tablet_bootstrap.cc:492] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:01.724220 2153 ts_tablet_manager.cc:1398] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:01.724459 2153 tablet_bootstrap.cc:492] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:01.724872 2038 tablet_bootstrap.cc:654] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
W20240502 01:59:01.726181 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:01.727391 2157 tablet_bootstrap.cc:492] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:01.727560 2157 ts_tablet_manager.cc:1367] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:01.728049 2157 raft_consensus.cc:348] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.728219 2157 raft_consensus.cc:374] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.728338 2157 raft_consensus.cc:729] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.728526 2157 consensus_queue.cc:260] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.728957 2157 ts_tablet_manager.cc:1398] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:01.729238 2153 tablet_bootstrap.cc:654] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.730494 2153 tablet_bootstrap.cc:492] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:01.730597 2153 ts_tablet_manager.cc:1367] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:01.731065 2153 raft_consensus.cc:348] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.731206 2153 raft_consensus.cc:374] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.731252 2153 raft_consensus.cc:729] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.731499 2153 consensus_queue.cc:260] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.731904 2153 ts_tablet_manager.cc:1398] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:01.734395 2038 tablet_bootstrap.cc:492] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:01.734490 2038 ts_tablet_manager.cc:1367] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:59:01.734876 2038 raft_consensus.cc:348] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.735008 2038 raft_consensus.cc:374] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.735039 2038 raft_consensus.cc:729] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.736150 2038 consensus_queue.cc:260] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.736510 2038 ts_tablet_manager.cc:1398] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:01.737790 2154 tablet_bootstrap.cc:492] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:01.737895 2154 ts_tablet_manager.cc:1367] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 01:59:01.738364 2154 raft_consensus.cc:348] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.738646 2154 raft_consensus.cc:374] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.738684 2154 raft_consensus.cc:729] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.738796 2154 consensus_queue.cc:260] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.739090 2154 ts_tablet_manager.cc:1398] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:01.739675 2158 tablet_bootstrap.cc:492] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:01.739769 2158 ts_tablet_manager.cc:1367] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 01:59:01.740244 2158 raft_consensus.cc:348] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.740352 2158 raft_consensus.cc:374] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.740402 2158 raft_consensus.cc:729] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.740521 2158 consensus_queue.cc:260] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.740833 2158 ts_tablet_manager.cc:1398] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:59:01.788399 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:01.809659 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:01.809850 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:01.812062 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet e0fd3df77d954a33bff826573ff79add
I20240502 01:59:01.812575 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd3df77d954a33bff826573ff79add with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:01 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:59:01.812688 24876 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd3df77d954a33bff826573ff79add with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:01 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:59:01.812866 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd3df77d954a33bff826573ff79add with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:01 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:59:01.812963 2166 tablet_replica.cc:330] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:59:01.813052 2167 tablet_replica.cc:330] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:59:01.813162 2166 raft_consensus.cc:2229] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:01.813225 2167 raft_consensus.cc:2229] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:01.813333 2167 raft_consensus.cc:2258] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.813360 2166 raft_consensus.cc:2258] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.813377 2168 tablet_replica.cc:330] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:59:01.813774 2168 raft_consensus.cc:2229] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:01.814004 2167 ts_tablet_manager.cc:1875] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:01.814157 2166 ts_tablet_manager.cc:1875] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:01.814407 2168 raft_consensus.cc:2258] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.815423 2168 ts_tablet_manager.cc:1875] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:01.816354 2167 ts_tablet_manager.cc:1888] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:01.816428 2167 log.cc:1196] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/e0fd3df77d954a33bff826573ff79add
I20240502 01:59:01.816828 2167 ts_tablet_manager.cc:1909] T e0fd3df77d954a33bff826573ff79add P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:59:01.817473 2168 ts_tablet_manager.cc:1888] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:01.817562 2168 log.cc:1196] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/e0fd3df77d954a33bff826573ff79add
I20240502 01:59:01.817556 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table_new_name [id=1ae341243e434fd0897276debbe56e51]) successfully deleted
I20240502 01:59:01.817847 2168 ts_tablet_manager.cc:1909] T e0fd3df77d954a33bff826573ff79add P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:59:01.818045 2166 ts_tablet_manager.cc:1888] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:01.818106 2166 log.cc:1196] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/e0fd3df77d954a33bff826573ff79add
I20240502 01:59:01.818348 2166 ts_tablet_manager.cc:1909] T e0fd3df77d954a33bff826573ff79add P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:59:01.818495 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table_new_name [id=1ae341243e434fd0897276debbe56e51]) successfully deleted
I20240502 01:59:01.818953 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet e0fd3df77d954a33bff826573ff79add (table rebalancer_dynamic_table_new_name [id=1ae341243e434fd0897276debbe56e51]) successfully deleted
I20240502 01:59:01.835664 1139 raft_consensus.cc:482] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:01.835773 1139 raft_consensus.cc:504] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.836174 1139 leader_election.cc:288] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:01.836512 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cec1ee29845d394d0d64486d974a7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:01.836495 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cec1ee29845d394d0d64486d974a7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:01.836642 24632 raft_consensus.cc:2454] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:01.836642 27577 raft_consensus.cc:2454] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:01.836926 27113 leader_election.cc:302] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:01.837395 1139 raft_consensus.cc:2781] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:01.837456 1139 raft_consensus.cc:482] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:01.837497 1139 raft_consensus.cc:3037] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.838443 1139 raft_consensus.cc:504] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.838788 1139 leader_election.cc:288] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:01.839756 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cec1ee29845d394d0d64486d974a7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:01.839864 24630 raft_consensus.cc:3037] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.839905 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2cec1ee29845d394d0d64486d974a7" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:01.840026 27577 raft_consensus.cc:3037] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.840695 24630 raft_consensus.cc:2454] T 1c2cec1ee29845d394d0d64486d974a7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:01.840785 27577 raft_consensus.cc:2454] T 1c2cec1ee29845d394d0d64486d974a7 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:01.841388 27112 leader_election.cc:302] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:01.841720 1139 raft_consensus.cc:2781] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:01.841820 1139 raft_consensus.cc:686] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:59:01.841993 1139 consensus_queue.cc:237] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:01.844717 24479 catalog_manager.cc:5484] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:01.881621 2121 raft_consensus.cc:482] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:01.881739 2121 raft_consensus.cc:504] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.882159 2121 leader_election.cc:288] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:01.882583 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e41255bbb47b38f00573acc07f235" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:01.882720 27446 raft_consensus.cc:2454] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:01.883029 24966 leader_election.cc:302] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:01.883244 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e41255bbb47b38f00573acc07f235" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:01.883380 24630 raft_consensus.cc:2454] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:01.883373 1891 raft_consensus.cc:2781] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:01.883438 1891 raft_consensus.cc:482] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:01.883473 1891 raft_consensus.cc:3037] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.884325 1891 raft_consensus.cc:504] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.884599 1891 leader_election.cc:288] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:01.884913 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e41255bbb47b38f00573acc07f235" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:01.885004 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e41255bbb47b38f00573acc07f235" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:01.885030 24630 raft_consensus.cc:3037] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.885089 27446 raft_consensus.cc:3037] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.885982 24630 raft_consensus.cc:2454] T 046e41255bbb47b38f00573acc07f235 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:01.885977 27446 raft_consensus.cc:2454] T 046e41255bbb47b38f00573acc07f235 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:01.886247 24966 leader_election.cc:302] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:01.886479 2046 raft_consensus.cc:2781] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:01.886543 2046 raft_consensus.cc:686] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:01.886660 2046 consensus_queue.cc:237] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:01.889128 24479 catalog_manager.cc:5484] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:01.905539 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:01.913753 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:01.919562 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 611af26e2eff4a508fee6231b5656ecc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]), partition=
I20240502 01:59:01.919850 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611af26e2eff4a508fee6231b5656ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.919909 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 611af26e2eff4a508fee6231b5656ecc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]), partition=
I20240502 01:59:01.920141 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611af26e2eff4a508fee6231b5656ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.922348 2153 tablet_bootstrap.cc:492] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:01.923228 2153 tablet_bootstrap.cc:654] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.924345 2153 tablet_bootstrap.cc:492] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:01.924422 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 611af26e2eff4a508fee6231b5656ecc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]), partition=
I20240502 01:59:01.924466 2153 ts_tablet_manager.cc:1367] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:01.924664 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611af26e2eff4a508fee6231b5656ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:01.924938 2153 raft_consensus.cc:348] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.925062 2153 raft_consensus.cc:374] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.925100 2153 raft_consensus.cc:729] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.925222 2153 consensus_queue.cc:260] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.925551 2153 ts_tablet_manager.cc:1398] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:01.926873 2154 tablet_bootstrap.cc:492] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:01.927713 2154 tablet_bootstrap.cc:654] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.928522 2037 tablet_bootstrap.cc:492] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:01.929322 2154 tablet_bootstrap.cc:492] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:01.929396 2037 tablet_bootstrap.cc:654] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:01.929404 2154 ts_tablet_manager.cc:1367] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:59:01.929852 2154 raft_consensus.cc:348] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.930004 2154 raft_consensus.cc:374] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.930055 2154 raft_consensus.cc:729] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.930191 2154 consensus_queue.cc:260] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.930488 2154 ts_tablet_manager.cc:1398] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:01.932461 2037 tablet_bootstrap.cc:492] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:01.932538 2037 ts_tablet_manager.cc:1367] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:59:01.932917 2037 raft_consensus.cc:348] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.933003 2037 raft_consensus.cc:374] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:01.933032 2037 raft_consensus.cc:729] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:01.933151 2037 consensus_queue.cc:260] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:01.933400 2037 ts_tablet_manager.cc:1398] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:59:01.938452 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:01.943240 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:01.950309 1738 raft_consensus.cc:482] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:01.950392 1738 raft_consensus.cc:504] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.950805 1738 leader_election.cc:288] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:01.951087 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7c40e2d1ad4c7d8dae6d95aad638bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:01.951143 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7c40e2d1ad4c7d8dae6d95aad638bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:01.951233 25031 raft_consensus.cc:2454] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:01.951244 24897 raft_consensus.cc:2454] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:01.951479 24701 leader_election.cc:302] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:01.951654 1738 raft_consensus.cc:2781] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:01.951709 1738 raft_consensus.cc:482] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:01.951735 1738 raft_consensus.cc:3037] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.952680 1738 raft_consensus.cc:504] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.952978 1738 leader_election.cc:288] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:01.953296 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7c40e2d1ad4c7d8dae6d95aad638bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:01.953311 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7c40e2d1ad4c7d8dae6d95aad638bf" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:01.953393 24897 raft_consensus.cc:3037] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.953403 25031 raft_consensus.cc:3037] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:01.954219 25031 raft_consensus.cc:2454] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:01.954360 24897 raft_consensus.cc:2454] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:01.954484 24698 leader_election.cc:302] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:01.954615 1738 raft_consensus.cc:2781] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:01.954684 1738 raft_consensus.cc:686] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:59:01.954778 1738 consensus_queue.cc:237] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:01.956634 24487 catalog_manager.cc:5484] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:02.101938 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:02.122579 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.225582 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.228209 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:02.304026 1139 consensus_queue.cc:1035] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.309492 1745 consensus_queue.cc:1035] T 1c2cec1ee29845d394d0d64486d974a7 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.317356 2046 raft_consensus.cc:482] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:02.317457 2046 raft_consensus.cc:504] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:02.317804 2046 leader_election.cc:288] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:02.318331 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611af26e2eff4a508fee6231b5656ecc" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:02.318488 24763 raft_consensus.cc:2454] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:02.318536 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611af26e2eff4a508fee6231b5656ecc" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:02.318653 27176 raft_consensus.cc:2454] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:02.318799 24966 leader_election.cc:302] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:02.319023 2046 raft_consensus.cc:2781] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:02.319065 2046 raft_consensus.cc:482] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:02.319089 2046 raft_consensus.cc:3037] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.319893 2046 raft_consensus.cc:504] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:02.320240 2046 leader_election.cc:288] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:02.320560 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611af26e2eff4a508fee6231b5656ecc" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:02.320657 27176 raft_consensus.cc:3037] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.320621 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611af26e2eff4a508fee6231b5656ecc" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:02.320729 24763 raft_consensus.cc:3037] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.321478 24763 raft_consensus.cc:2454] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:02.321599 27176 raft_consensus.cc:2454] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:02.321904 24965 leader_election.cc:302] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:02.322104 2046 raft_consensus.cc:2781] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:02.322181 2046 raft_consensus.cc:686] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:02.322309 2046 consensus_queue.cc:237] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:02.324617 24487 catalog_manager.cc:5484] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:02.334369 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.353418 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.382141 1738 consensus_queue.cc:1035] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:02.386080 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.388509 1736 consensus_queue.cc:1035] T 8f7c40e2d1ad4c7d8dae6d95aad638bf P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.441649 2046 consensus_queue.cc:1035] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.446691 2046 consensus_queue.cc:1035] T 046e41255bbb47b38f00573acc07f235 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.466303 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:02.472164 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 8dae3ac764a344cd8fe1304726b59a8d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.472465 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dae3ac764a344cd8fe1304726b59a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.472431 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 8dae3ac764a344cd8fe1304726b59a8d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.473098 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dae3ac764a344cd8fe1304726b59a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.473067 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 8dae3ac764a344cd8fe1304726b59a8d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.473309 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dae3ac764a344cd8fe1304726b59a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.475574 2271 tablet_bootstrap.cc:492] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:02.475567 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 6081fc4d64f3442fb3b34adfd74d2d5f (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.475777 27558 tablet_service.cc:1467] Processing CreateTablet for tablet d60c049ab8bd42ef9ab74a2d5a7cbf76 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.475855 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6081fc4d64f3442fb3b34adfd74d2d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.476014 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c049ab8bd42ef9ab74a2d5a7cbf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.476135 27423 tablet_service.cc:1467] Processing CreateTablet for tablet d60c049ab8bd42ef9ab74a2d5a7cbf76 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.476210 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 6081fc4d64f3442fb3b34adfd74d2d5f (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.476328 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c049ab8bd42ef9ab74a2d5a7cbf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.476414 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6081fc4d64f3442fb3b34adfd74d2d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.478538 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 6081fc4d64f3442fb3b34adfd74d2d5f (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.478754 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6081fc4d64f3442fb3b34adfd74d2d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.478946 2271 tablet_bootstrap.cc:654] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.479231 24876 tablet_service.cc:1467] Processing CreateTablet for tablet d60c049ab8bd42ef9ab74a2d5a7cbf76 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c40b8f9a549b4387ac27512583ef4b31]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:02.479441 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c049ab8bd42ef9ab74a2d5a7cbf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.481720 24487 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:02.482798 2273 tablet_bootstrap.cc:492] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:02.483397 2274 tablet_bootstrap.cc:492] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:02.484405 2275 tablet_bootstrap.cc:492] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:02.486146 2272 tablet_bootstrap.cc:492] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:02.487207 2273 tablet_bootstrap.cc:654] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.487342 2275 tablet_bootstrap.cc:654] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.489746 2272 tablet_bootstrap.cc:654] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.490239 2274 tablet_bootstrap.cc:654] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.491784 27176 raft_consensus.cc:1261] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:02.492090 2275 tablet_bootstrap.cc:492] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:02.492157 2121 consensus_queue.cc:1035] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.492506 2275 ts_tablet_manager.cc:1367] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:59:02.492962 24763 raft_consensus.cc:1261] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:02.493073 2275 raft_consensus.cc:348] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.493397 2275 raft_consensus.cc:374] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.493482 2275 raft_consensus.cc:729] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.493645 2275 consensus_queue.cc:260] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.494016 2275 ts_tablet_manager.cc:1398] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:02.494271 2275 tablet_bootstrap.cc:492] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:02.494956 2274 tablet_bootstrap.cc:492] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:02.494925 2121 consensus_queue.cc:1035] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.495123 2274 ts_tablet_manager.cc:1367] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:59:02.495556 2275 tablet_bootstrap.cc:654] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.495607 24487 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]). Expected version 1 got 0
I20240502 01:59:02.495599 2274 raft_consensus.cc:348] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.495690 2274 raft_consensus.cc:374] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.495724 2274 raft_consensus.cc:729] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.495831 2274 consensus_queue.cc:260] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.496161 2274 ts_tablet_manager.cc:1398] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:02.497115 2272 tablet_bootstrap.cc:492] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:02.497380 2272 ts_tablet_manager.cc:1367] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:59:02.497891 2272 raft_consensus.cc:348] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.498107 2272 raft_consensus.cc:374] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.498155 2272 raft_consensus.cc:729] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.498271 2272 consensus_queue.cc:260] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.498572 2272 ts_tablet_manager.cc:1398] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:02.498713 2272 tablet_bootstrap.cc:492] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:02.498984 24482 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]). Expected version 1 got 0
I20240502 01:59:02.499617 2272 tablet_bootstrap.cc:654] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.500636 2272 tablet_bootstrap.cc:492] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:02.500640 2271 tablet_bootstrap.cc:492] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:02.500725 2271 ts_tablet_manager.cc:1367] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240502 01:59:02.500798 2272 ts_tablet_manager.cc:1367] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:02.501209 2272 raft_consensus.cc:348] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.501209 2271 raft_consensus.cc:348] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.501302 2271 raft_consensus.cc:374] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.501338 2271 raft_consensus.cc:729] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.501471 2272 raft_consensus.cc:374] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.501458 2271 consensus_queue.cc:260] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.501570 2272 raft_consensus.cc:729] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.501694 2266 mvcc.cc:205] Tried to move back new op lower bound from 7023063623642939392 to 7023063622953578496. Current Snapshot: MvccSnapshot[applied={T|T < 7023063623642939392}]
I20240502 01:59:02.501746 2271 ts_tablet_manager.cc:1398] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:02.501902 2271 tablet_bootstrap.cc:492] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:02.502918 2271 tablet_bootstrap.cc:654] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.503355 2272 consensus_queue.cc:260] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.503489 24487 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]). Expected version 1 got 0
I20240502 01:59:02.503700 2272 ts_tablet_manager.cc:1398] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:02.504631 24482 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]). Expected version 1 got 0
I20240502 01:59:02.505743 2286 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:02.506045 2286 tablet.cc:1620] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:59:02.507665 2273 tablet_bootstrap.cc:492] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:02.507778 2273 ts_tablet_manager.cc:1367] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240502 01:59:02.508004 24487 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]). Expected version 1 got 0
I20240502 01:59:02.508325 24479 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f]). Expected version 1 got 0
I20240502 01:59:02.508471 2273 raft_consensus.cc:348] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.508658 2273 raft_consensus.cc:374] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.508730 2273 raft_consensus.cc:729] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.508879 2273 consensus_queue.cc:260] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.509187 2273 ts_tablet_manager.cc:1398] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:02.509418 2285 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:02.509742 2285 tablet.cc:1620] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:02.510532 2275 tablet_bootstrap.cc:492] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:02.510623 2275 ts_tablet_manager.cc:1367] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:59:02.511042 2275 raft_consensus.cc:348] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.511154 2275 raft_consensus.cc:374] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.511181 2275 raft_consensus.cc:729] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.511350 2275 consensus_queue.cc:260] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.511673 2275 ts_tablet_manager.cc:1398] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:02.512044 2288 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:02.512385 2288 tablet.cc:1620] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:02.514837 2271 tablet_bootstrap.cc:492] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:02.514916 2271 ts_tablet_manager.cc:1367] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 01:59:02.515331 2271 raft_consensus.cc:348] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.515483 2271 raft_consensus.cc:374] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.515554 2271 raft_consensus.cc:729] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.515736 2271 consensus_queue.cc:260] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.516029 2271 ts_tablet_manager.cc:1398] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:02.516191 2271 tablet_bootstrap.cc:492] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:02.517392 2271 tablet_bootstrap.cc:654] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.519402 24487 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f] alter complete (version 1)
I20240502 01:59:02.520877 2271 tablet_bootstrap.cc:492] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:02.520956 2271 ts_tablet_manager.cc:1367] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:59:02.521315 2271 raft_consensus.cc:348] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.521441 2271 raft_consensus.cc:374] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.521512 2271 raft_consensus.cc:729] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.521638 2271 consensus_queue.cc:260] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.521885 2271 ts_tablet_manager.cc:1398] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:02.535332 1139 raft_consensus.cc:482] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:02.535449 1139 raft_consensus.cc:504] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.535784 1139 leader_election.cc:288] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:02.536229 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6081fc4d64f3442fb3b34adfd74d2d5f" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:02.536249 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6081fc4d64f3442fb3b34adfd74d2d5f" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:02.536372 27446 raft_consensus.cc:2454] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:02.536387 27578 raft_consensus.cc:2454] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:02.536626 2261 raft_consensus.cc:482] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:02.536671 27113 leader_election.cc:302] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:02.536729 2261 raft_consensus.cc:504] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.536826 1139 raft_consensus.cc:2781] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:02.536880 1139 raft_consensus.cc:482] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:02.536916 1139 raft_consensus.cc:3037] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.537106 2261 leader_election.cc:288] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:02.537595 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c049ab8bd42ef9ab74a2d5a7cbf76" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:02.537700 27446 raft_consensus.cc:2454] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:02.537890 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c049ab8bd42ef9ab74a2d5a7cbf76" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:02.537978 1139 raft_consensus.cc:504] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.538020 27578 raft_consensus.cc:2454] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:02.538055 24833 leader_election.cc:302] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:02.538285 1139 leader_election.cc:288] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:02.538472 2261 raft_consensus.cc:2781] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:02.538532 2261 raft_consensus.cc:482] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:02.538566 2261 raft_consensus.cc:3037] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.538683 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6081fc4d64f3442fb3b34adfd74d2d5f" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:02.538735 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6081fc4d64f3442fb3b34adfd74d2d5f" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:02.538775 27446 raft_consensus.cc:3037] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.538832 27578 raft_consensus.cc:3037] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.539546 27578 raft_consensus.cc:2454] T 6081fc4d64f3442fb3b34adfd74d2d5f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:02.539564 2261 raft_consensus.cc:504] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.539630 27446 raft_consensus.cc:2454] T 6081fc4d64f3442fb3b34adfd74d2d5f P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:02.539810 27113 leader_election.cc:302] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:02.539888 2261 leader_election.cc:288] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:02.540161 1139 raft_consensus.cc:2781] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:02.540241 1139 raft_consensus.cc:686] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:59:02.540369 1139 consensus_queue.cc:237] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:02.540673 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c049ab8bd42ef9ab74a2d5a7cbf76" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:02.540774 27446 raft_consensus.cc:3037] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.541637 27446 raft_consensus.cc:2454] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:02.542388 24833 leader_election.cc:302] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:02.542547 2261 raft_consensus.cc:2781] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:02.542618 2261 raft_consensus.cc:686] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:02.542738 2261 consensus_queue.cc:237] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.542833 24482 catalog_manager.cc:5484] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:02.543510 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c049ab8bd42ef9ab74a2d5a7cbf76" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:02.543605 27578 raft_consensus.cc:3037] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.544481 27578 raft_consensus.cc:2454] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:02.545729 24487 catalog_manager.cc:5484] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:02.594872 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.595221 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.602195 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.607605 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.612816 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:02.614586 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.622812 1926 raft_consensus.cc:482] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:02.622905 1926 raft_consensus.cc:504] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.623230 1926 leader_election.cc:288] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:02.623510 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dae3ac764a344cd8fe1304726b59a8d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:02.623637 24763 raft_consensus.cc:2454] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:02.623641 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dae3ac764a344cd8fe1304726b59a8d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:02.623775 24897 raft_consensus.cc:2454] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:02.623872 27380 leader_election.cc:302] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:02.624140 1926 raft_consensus.cc:2781] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:02.624192 1926 raft_consensus.cc:482] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:02.624217 1926 raft_consensus.cc:3037] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.625102 1926 raft_consensus.cc:504] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.625236 24487 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:02.625428 1926 leader_election.cc:288] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:02.625780 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dae3ac764a344cd8fe1304726b59a8d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:02.625883 24763 raft_consensus.cc:3037] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.626005 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dae3ac764a344cd8fe1304726b59a8d" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:02.626101 24897 raft_consensus.cc:3037] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:02.626884 24763 raft_consensus.cc:2454] T 8dae3ac764a344cd8fe1304726b59a8d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:02.627014 24897 raft_consensus.cc:2454] T 8dae3ac764a344cd8fe1304726b59a8d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:02.627266 27380 leader_election.cc:302] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
W20240502 01:59:02.627394 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.627602 2228 raft_consensus.cc:2781] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:02.627755 2228 raft_consensus.cc:686] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:02.627880 2228 consensus_queue.cc:237] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.630504 24487 catalog_manager.cc:5484] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:02.633260 2285 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:02.633534 2285 tablet.cc:1620] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:02.633627 2286 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:02.633891 2286 tablet.cc:1620] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:59:02.634236 2288 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:02.634492 2288 tablet.cc:1620] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:02.639281 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=b92d78db4a544a9789d25d848f4d251f] alter complete (version 2)
W20240502 01:59:02.661118 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:02.678004 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.740653 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:02.745879 2285 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:02.746873 2286 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:02.746925 2288 tablet.cc:1722] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:02.752359 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=b92d78db4a544a9789d25d848f4d251f] alter complete (version 3)
W20240502 01:59:02.761168 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.785671 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:02.815248 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.854877 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:02.855036 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:02.857126 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 611af26e2eff4a508fee6231b5656ecc
I20240502 01:59:02.857592 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet 611af26e2eff4a508fee6231b5656ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:02 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:59:02.857893 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 611af26e2eff4a508fee6231b5656ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:02 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:59:02.858042 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 611af26e2eff4a508fee6231b5656ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:02 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:02.858227 2345 tablet_replica.cc:330] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:59:02.858381 2346 tablet_replica.cc:330] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:02.858445 2344 tablet_replica.cc:330] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:59:02.858443 2345 raft_consensus.cc:2229] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:02.858536 2346 raft_consensus.cc:2229] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:02.858561 2345 raft_consensus.cc:2258] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:02.858618 2344 raft_consensus.cc:2229] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:02.858763 2344 raft_consensus.cc:2258] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:02.859457 2345 ts_tablet_manager.cc:1875] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:02.859601 2344 ts_tablet_manager.cc:1875] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:02.860002 2346 raft_consensus.cc:2258] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:02.860608 2346 ts_tablet_manager.cc:1875] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:02.861868 2345 ts_tablet_manager.cc:1888] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:02.862021 2345 log.cc:1196] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/611af26e2eff4a508fee6231b5656ecc
I20240502 01:59:02.862363 2345 ts_tablet_manager.cc:1909] T 611af26e2eff4a508fee6231b5656ecc P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:59:02.862367 2344 ts_tablet_manager.cc:1888] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:02.862427 2344 log.cc:1196] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/611af26e2eff4a508fee6231b5656ecc
I20240502 01:59:02.862700 2344 ts_tablet_manager.cc:1909] T 611af26e2eff4a508fee6231b5656ecc P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:59:02.862808 2346 ts_tablet_manager.cc:1888] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:02.862882 2346 log.cc:1196] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/611af26e2eff4a508fee6231b5656ecc
I20240502 01:59:02.863032 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table_new_name [id=b92d78db4a544a9789d25d848f4d251f]) successfully deleted
I20240502 01:59:02.863229 2346 ts_tablet_manager.cc:1909] T 611af26e2eff4a508fee6231b5656ecc P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:02.863337 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table_new_name [id=b92d78db4a544a9789d25d848f4d251f]) successfully deleted
I20240502 01:59:02.863771 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 611af26e2eff4a508fee6231b5656ecc (table rebalancer_dynamic_table_new_name [id=b92d78db4a544a9789d25d848f4d251f]) successfully deleted
I20240502 01:59:02.921603 1800 consensus_queue.cc:1035] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.921717 2347 consensus_queue.cc:1035] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.925388 2261 consensus_queue.cc:1035] T d60c049ab8bd42ef9ab74a2d5a7cbf76 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.930125 1139 consensus_queue.cc:1035] T 6081fc4d64f3442fb3b34adfd74d2d5f P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:02.958596 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:02.963178 27157 tablet_service.cc:1467] Processing CreateTablet for tablet d9b4536d69bc46b0972571acd926ea4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb]), partition=
I20240502 01:59:02.963178 24610 tablet_service.cc:1467] Processing CreateTablet for tablet d9b4536d69bc46b0972571acd926ea4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb]), partition=
I20240502 01:59:02.963299 24876 tablet_service.cc:1467] Processing CreateTablet for tablet d9b4536d69bc46b0972571acd926ea4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb]), partition=
I20240502 01:59:02.963416 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b4536d69bc46b0972571acd926ea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.963431 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b4536d69bc46b0972571acd926ea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.963517 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b4536d69bc46b0972571acd926ea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:02.965144 2272 tablet_bootstrap.cc:492] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:02.965497 2274 tablet_bootstrap.cc:492] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:02.966037 2272 tablet_bootstrap.cc:654] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.966389 2274 tablet_bootstrap.cc:654] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.966833 2355 tablet_bootstrap.cc:492] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:02.967780 2355 tablet_bootstrap.cc:654] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:02.967840 2274 tablet_bootstrap.cc:492] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:02.967926 2274 ts_tablet_manager.cc:1367] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
W20240502 01:59:02.968164 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:02.968376 2274 raft_consensus.cc:348] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.968482 2274 raft_consensus.cc:374] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.968511 2274 raft_consensus.cc:729] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.968660 2272 tablet_bootstrap.cc:492] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:02.968631 2274 consensus_queue.cc:260] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.968727 2272 ts_tablet_manager.cc:1367] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:59:02.968899 2355 tablet_bootstrap.cc:492] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:02.968906 2274 ts_tablet_manager.cc:1398] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:02.968958 2355 ts_tablet_manager.cc:1367] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:02.969130 2272 raft_consensus.cc:348] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.969224 2272 raft_consensus.cc:374] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.969261 2272 raft_consensus.cc:729] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.969305 2355 raft_consensus.cc:348] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.969372 2272 consensus_queue.cc:260] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.969440 2355 raft_consensus.cc:374] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:02.969480 2355 raft_consensus.cc:729] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:02.969588 2355 consensus_queue.cc:260] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:02.969672 2272 ts_tablet_manager.cc:1398] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:02.970592 2355 ts_tablet_manager.cc:1398] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 01:59:02.995286 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:03.010749 2317 consensus_queue.cc:1035] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.015478 1926 consensus_queue.cc:1035] T 8dae3ac764a344cd8fe1304726b59a8d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:03.055210 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:03.074767 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:03.146739 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:03.151901 2195 raft_consensus.cc:482] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:03.152010 2195 raft_consensus.cc:504] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.152397 2195 leader_election.cc:288] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:03.152673 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b4536d69bc46b0972571acd926ea4b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:03.152813 24897 raft_consensus.cc:2454] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:03.153225 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b4536d69bc46b0972571acd926ea4b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:03.153350 27176 raft_consensus.cc:2454] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:03.153368 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 096823fbfa5d44fda29590a52ef7c9f7 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.153645 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096823fbfa5d44fda29590a52ef7c9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.153712 24566 leader_election.cc:302] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:03.153874 2195 raft_consensus.cc:2781] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:03.153959 2195 raft_consensus.cc:482] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:03.153996 2195 raft_consensus.cc:3037] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.154188 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 096823fbfa5d44fda29590a52ef7c9f7 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.154474 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096823fbfa5d44fda29590a52ef7c9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.154657 25010 tablet_service.cc:1467] Processing CreateTablet for tablet aedface7dd7c44ab8b5e510c633f951b (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.154888 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedface7dd7c44ab8b5e510c633f951b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.154922 2195 raft_consensus.cc:504] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.155259 2195 leader_election.cc:288] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:03.155886 2355 tablet_bootstrap.cc:492] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:03.156069 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 096823fbfa5d44fda29590a52ef7c9f7 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.156275 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096823fbfa5d44fda29590a52ef7c9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.156805 2355 tablet_bootstrap.cc:654] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.156924 27157 tablet_service.cc:1467] Processing CreateTablet for tablet aedface7dd7c44ab8b5e510c633f951b (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.157167 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedface7dd7c44ab8b5e510c633f951b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.157310 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b4536d69bc46b0972571acd926ea4b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:03.157397 27176 raft_consensus.cc:3037] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.157950 2355 tablet_bootstrap.cc:492] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:03.158037 2355 ts_tablet_manager.cc:1367] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:03.158249 27176 raft_consensus.cc:2454] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:03.158468 2355 raft_consensus.cc:348] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.158517 24566 leader_election.cc:302] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:03.158555 2355 raft_consensus.cc:374] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.158586 2355 raft_consensus.cc:729] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.158694 2355 consensus_queue.cc:260] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.158864 2078 raft_consensus.cc:2781] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:03.158942 2078 raft_consensus.cc:686] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:03.159062 2355 ts_tablet_manager.cc:1398] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.159070 2078 consensus_queue.cc:237] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.159209 2274 tablet_bootstrap.cc:492] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:03.160122 2274 tablet_bootstrap.cc:654] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.160425 2387 tablet_bootstrap.cc:492] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:03.161197 2274 tablet_bootstrap.cc:492] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:03.161285 2274 ts_tablet_manager.cc:1367] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:03.161293 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 896090c872314da4a317cd155f9807bd (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.161556 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896090c872314da4a317cd155f9807bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.161701 2274 raft_consensus.cc:348] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.161806 2274 raft_consensus.cc:374] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.161839 2274 raft_consensus.cc:729] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.161861 2387 tablet_bootstrap.cc:654] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.162009 2274 consensus_queue.cc:260] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.162276 2274 ts_tablet_manager.cc:1398] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:03.162612 24744 tablet_service.cc:1467] Processing CreateTablet for tablet aedface7dd7c44ab8b5e510c633f951b (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.162840 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aedface7dd7c44ab8b5e510c633f951b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.163992 2393 tablet_bootstrap.cc:492] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:03.164152 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 896090c872314da4a317cd155f9807bd (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.164379 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896090c872314da4a317cd155f9807bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.165268 2393 tablet_bootstrap.cc:654] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.165665 24487 catalog_manager.cc:5484] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:03.166584 2393 tablet_bootstrap.cc:492] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:03.166680 2393 ts_tablet_manager.cc:1367] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:03.166797 2386 tablet_bootstrap.cc:492] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:03.167187 2393 raft_consensus.cc:348] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.167340 2393 raft_consensus.cc:374] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.167398 2393 raft_consensus.cc:729] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.167562 2393 consensus_queue.cc:260] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.167910 2393 ts_tablet_manager.cc:1398] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.167919 2386 tablet_bootstrap.cc:654] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.168972 2386 tablet_bootstrap.cc:492] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:03.168975 2387 tablet_bootstrap.cc:492] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:03.169060 2387 ts_tablet_manager.cc:1367] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:03.169101 2386 ts_tablet_manager.cc:1367] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:03.169509 2387 raft_consensus.cc:348] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.169602 2387 raft_consensus.cc:374] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.169632 2387 raft_consensus.cc:729] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.169637 2386 raft_consensus.cc:348] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.169742 2387 consensus_queue.cc:260] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.169785 2386 raft_consensus.cc:374] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.169826 2386 raft_consensus.cc:729] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.169971 2386 consensus_queue.cc:260] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.170045 2387 ts_tablet_manager.cc:1398] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.170181 2387 tablet_bootstrap.cc:492] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:03.170482 2386 ts_tablet_manager.cc:1398] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.170660 2386 tablet_bootstrap.cc:492] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:03.170996 2387 tablet_bootstrap.cc:654] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.171561 2386 tablet_bootstrap.cc:654] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.172130 2387 tablet_bootstrap.cc:492] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:03.172215 2387 ts_tablet_manager.cc:1367] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:03.172636 2387 raft_consensus.cc:348] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.172724 2387 raft_consensus.cc:374] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.172752 2387 raft_consensus.cc:729] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.172860 2387 consensus_queue.cc:260] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.173099 2387 ts_tablet_manager.cc:1398] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.173791 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b4536d69bc46b0972571acd926ea4b" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:03.173910 24897 raft_consensus.cc:3037] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.174896 24897 raft_consensus.cc:2454] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:03.175031 1891 raft_consensus.cc:482] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:03.175107 1891 raft_consensus.cc:504] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.175416 1891 leader_election.cc:288] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:03.176153 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedface7dd7c44ab8b5e510c633f951b" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
W20240502 01:59:03.176541 24966 leader_election.cc:341] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:59:03.176712 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedface7dd7c44ab8b5e510c633f951b" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:03.176827 27176 raft_consensus.cc:2454] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:03.177141 24965 leader_election.cc:302] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters: 92cb785c165749eca49af5784756461c
I20240502 01:59:03.177515 2046 raft_consensus.cc:2781] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:03.177569 2046 raft_consensus.cc:482] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:03.177606 2046 raft_consensus.cc:3037] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.178567 2046 raft_consensus.cc:504] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.178879 2046 leader_election.cc:288] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:03.179407 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 896090c872314da4a317cd155f9807bd (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=014dd9abd97c4cb68071f749041a3c0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:03.179670 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896090c872314da4a317cd155f9807bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.179708 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedface7dd7c44ab8b5e510c633f951b" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
W20240502 01:59:03.180126 24966 leader_election.cc:341] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:59:03.180392 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedface7dd7c44ab8b5e510c633f951b" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:03.180506 27176 raft_consensus.cc:3037] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.181445 27176 raft_consensus.cc:2454] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:03.181695 2272 tablet_bootstrap.cc:492] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:03.181725 24965 leader_election.cc:302] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters: 92cb785c165749eca49af5784756461c
I20240502 01:59:03.181921 1891 raft_consensus.cc:2781] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:03.182024 1891 raft_consensus.cc:686] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:03.182137 1891 consensus_queue.cc:237] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.182612 2272 tablet_bootstrap.cc:654] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.184331 24481 catalog_manager.cc:5484] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:03.185364 2272 tablet_bootstrap.cc:492] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:03.185432 2272 ts_tablet_manager.cc:1367] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:59:03.185803 2272 raft_consensus.cc:348] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.185883 2272 raft_consensus.cc:374] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.185917 2272 raft_consensus.cc:729] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.186045 2272 consensus_queue.cc:260] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.186285 2272 ts_tablet_manager.cc:1398] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:03.187382 2386 tablet_bootstrap.cc:492] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:03.187522 2386 ts_tablet_manager.cc:1367] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:59:03.188167 2386 raft_consensus.cc:348] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.188266 2386 raft_consensus.cc:374] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.188295 2386 raft_consensus.cc:729] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.188407 2386 consensus_queue.cc:260] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.188692 2386 ts_tablet_manager.cc:1398] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.188897 2386 tablet_bootstrap.cc:492] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:03.189733 2386 tablet_bootstrap.cc:654] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.192142 2386 tablet_bootstrap.cc:492] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:03.192262 2386 ts_tablet_manager.cc:1367] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:03.193128 2386 raft_consensus.cc:348] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.193269 2386 raft_consensus.cc:374] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.193313 2386 raft_consensus.cc:729] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.193413 2386 consensus_queue.cc:260] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.193650 2386 ts_tablet_manager.cc:1398] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:59:03.253402 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:03.261943 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:03.277281 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:03.307705 2046 raft_consensus.cc:482] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:03.307808 2046 raft_consensus.cc:504] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.308183 2046 leader_election.cc:288] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:03.308542 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096823fbfa5d44fda29590a52ef7c9f7" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:03.308542 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096823fbfa5d44fda29590a52ef7c9f7" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:03.308688 24630 raft_consensus.cc:2454] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:03.308688 24763 raft_consensus.cc:2454] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:03.308986 24966 leader_election.cc:302] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:03.309180 2046 raft_consensus.cc:2781] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:03.309237 2046 raft_consensus.cc:482] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:03.309279 2046 raft_consensus.cc:3037] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.310191 2046 raft_consensus.cc:504] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.310562 2046 leader_election.cc:288] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:03.310832 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096823fbfa5d44fda29590a52ef7c9f7" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:03.310904 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096823fbfa5d44fda29590a52ef7c9f7" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:03.310961 24763 raft_consensus.cc:3037] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.311020 24630 raft_consensus.cc:3037] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.311983 24763 raft_consensus.cc:2454] T 096823fbfa5d44fda29590a52ef7c9f7 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:03.311985 24630 raft_consensus.cc:2454] T 096823fbfa5d44fda29590a52ef7c9f7 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:03.312263 24966 leader_election.cc:302] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:03.312407 2046 raft_consensus.cc:2781] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:03.312498 2046 raft_consensus.cc:686] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:03.312624 2046 consensus_queue.cc:237] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.315644 24482 catalog_manager.cc:5484] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:03.323318 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:03.327131 24897 raft_consensus.cc:1261] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:03.327247 27176 raft_consensus.cc:1261] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:03.327524 2195 consensus_queue.cc:1035] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.327680 2389 consensus_queue.cc:1035] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.330610 2432 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:03.330955 2432 tablet.cc:1620] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:03.333108 24481 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet d9b4536d69bc46b0972571acd926ea4b (table rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb]). Expected version 1 got 0
I20240502 01:59:03.335613 2433 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:03.335875 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb] alter complete (version 1)
I20240502 01:59:03.336023 2433 tablet.cc:1620] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:03.337672 2434 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:03.338030 2434 tablet.cc:1620] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:03.339046 24482 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet d9b4536d69bc46b0972571acd926ea4b (table rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb]). Expected version 1 got 0
I20240502 01:59:03.440958 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:03.445978 2432 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:03.446300 2432 tablet.cc:1620] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:03.446417 2434 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:03.446686 2434 tablet.cc:1620] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:03.449193 2433 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:03.449476 2433 tablet.cc:1620] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:03.452018 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=5b30884fb2264a72b11d1bab683173eb] alter complete (version 2)
I20240502 01:59:03.484284 2400 raft_consensus.cc:482] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:03.484369 2400 raft_consensus.cc:504] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.484673 2400 leader_election.cc:288] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:03.485189 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896090c872314da4a317cd155f9807bd" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:03.485198 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896090c872314da4a317cd155f9807bd" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:03.485319 24763 raft_consensus.cc:2454] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:03.485330 27578 raft_consensus.cc:2454] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:03.485612 24833 leader_election.cc:302] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:03.485962 2400 raft_consensus.cc:2781] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:03.486018 2400 raft_consensus.cc:482] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:03.486050 2400 raft_consensus.cc:3037] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.487028 2400 raft_consensus.cc:504] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.487337 2400 leader_election.cc:288] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:03.489107 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896090c872314da4a317cd155f9807bd" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:03.489214 27578 raft_consensus.cc:3037] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.489534 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896090c872314da4a317cd155f9807bd" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:03.489641 24763 raft_consensus.cc:3037] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.490173 27578 raft_consensus.cc:2454] T 896090c872314da4a317cd155f9807bd P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:03.490451 24833 leader_election.cc:302] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:03.490566 24763 raft_consensus.cc:2454] T 896090c872314da4a317cd155f9807bd P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:03.490607 2400 raft_consensus.cc:2781] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:03.490682 2400 raft_consensus.cc:686] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:03.490805 2400 consensus_queue.cc:237] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:03.493011 24486 catalog_manager.cc:5484] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:03.555626 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:03.560333 2432 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:03.560765 2433 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:03.560822 2434 tablet.cc:1722] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:03.566308 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=5b30884fb2264a72b11d1bab683173eb] alter complete (version 3)
I20240502 01:59:03.636777 1736 raft_consensus.cc:482] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:03.636902 1736 raft_consensus.cc:504] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:03.637303 1736 leader_election.cc:288] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:03.637657 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedface7dd7c44ab8b5e510c633f951b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:03.637710 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aedface7dd7c44ab8b5e510c633f951b" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:03.637842 27176 raft_consensus.cc:2379] T aedface7dd7c44ab8b5e510c633f951b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92cb785c165749eca49af5784756461c in current term 1: Already voted for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in this term.
I20240502 01:59:03.638152 24700 leader_election.cc:302] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92cb785c165749eca49af5784756461c; no voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657
I20240502 01:59:03.638336 1736 raft_consensus.cc:3037] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.639192 1736 raft_consensus.cc:2726] T aedface7dd7c44ab8b5e510c633f951b P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:59:03.668430 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:03.668588 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:03.670845 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet d9b4536d69bc46b0972571acd926ea4b
I20240502 01:59:03.671391 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b4536d69bc46b0972571acd926ea4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:03 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:59:03.671659 27157 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b4536d69bc46b0972571acd926ea4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:03 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:59:03.671682 2462 tablet_replica.cc:330] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:59:03.671829 2462 raft_consensus.cc:2229] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:03.671981 2463 tablet_replica.cc:330] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:59:03.672035 2462 raft_consensus.cc:2258] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:03.672071 24876 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b4536d69bc46b0972571acd926ea4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:03 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:59:03.672315 2463 raft_consensus.cc:2229] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:03.672353 2464 tablet_replica.cc:330] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:59:03.672462 2463 raft_consensus.cc:2258] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:03.672485 2464 raft_consensus.cc:2229] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:03.672605 2464 raft_consensus.cc:2258] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:03.672863 2462 ts_tablet_manager.cc:1875] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:03.673017 2463 ts_tablet_manager.cc:1875] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:03.673223 2464 ts_tablet_manager.cc:1875] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:03.675376 2463 ts_tablet_manager.cc:1888] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:03.675489 2463 log.cc:1196] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/d9b4536d69bc46b0972571acd926ea4b
I20240502 01:59:03.675568 2462 ts_tablet_manager.cc:1888] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:03.675653 2462 log.cc:1196] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/d9b4536d69bc46b0972571acd926ea4b
I20240502 01:59:03.675720 2464 ts_tablet_manager.cc:1888] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:03.675786 2463 ts_tablet_manager.cc:1909] T d9b4536d69bc46b0972571acd926ea4b P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:59:03.675832 2464 log.cc:1196] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/d9b4536d69bc46b0972571acd926ea4b
I20240502 01:59:03.675959 2462 ts_tablet_manager.cc:1909] T d9b4536d69bc46b0972571acd926ea4b P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:59:03.676199 2464 ts_tablet_manager.cc:1909] T d9b4536d69bc46b0972571acd926ea4b P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:59:03.676530 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet d9b4536d69bc46b0972571acd926ea4b (table rebalancer_dynamic_table_new_name [id=5b30884fb2264a72b11d1bab683173eb]) successfully deleted
I20240502 01:59:03.676692 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet d9b4536d69bc46b0972571acd926ea4b (table rebalancer_dynamic_table_new_name [id=5b30884fb2264a72b11d1bab683173eb]) successfully deleted
I20240502 01:59:03.676934 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet d9b4536d69bc46b0972571acd926ea4b (table rebalancer_dynamic_table_new_name [id=5b30884fb2264a72b11d1bab683173eb]) successfully deleted
I20240502 01:59:03.715859 2046 consensus_queue.cc:1035] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.721972 1891 consensus_queue.cc:1035] T aedface7dd7c44ab8b5e510c633f951b P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:03.724395 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:03.727810 1891 consensus_queue.cc:1035] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.732434 2046 consensus_queue.cc:1035] T 096823fbfa5d44fda29590a52ef7c9f7 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:03.733439 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:03.764791 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:03.772486 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:03.778609 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 793f76c356c24b8aa5b0c4ac92beabc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b]), partition=
I20240502 01:59:03.778692 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 793f76c356c24b8aa5b0c4ac92beabc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b]), partition=
I20240502 01:59:03.778915 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f76c356c24b8aa5b0c4ac92beabc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.778909 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 793f76c356c24b8aa5b0c4ac92beabc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b]), partition=
I20240502 01:59:03.778952 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f76c356c24b8aa5b0c4ac92beabc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.779127 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793f76c356c24b8aa5b0c4ac92beabc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:03.781091 2498 tablet_bootstrap.cc:492] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:03.781510 2500 tablet_bootstrap.cc:492] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:03.782408 2500 tablet_bootstrap.cc:654] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.782590 2498 tablet_bootstrap.cc:654] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.783095 2499 tablet_bootstrap.cc:492] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:03.783969 2498 tablet_bootstrap.cc:492] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:03.784108 2498 ts_tablet_manager.cc:1367] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:03.784433 2500 tablet_bootstrap.cc:492] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:03.784018 2499 tablet_bootstrap.cc:654] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:03.784657 2500 ts_tablet_manager.cc:1367] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:03.784639 2498 raft_consensus.cc:348] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.784806 2498 raft_consensus.cc:374] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.784860 2498 raft_consensus.cc:729] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.785018 2498 consensus_queue.cc:260] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.785085 2500 raft_consensus.cc:348] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.785177 2500 raft_consensus.cc:374] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.785207 2500 raft_consensus.cc:729] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.785574 2498 ts_tablet_manager.cc:1398] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:03.786729 2500 consensus_queue.cc:260] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.787058 2500 ts_tablet_manager.cc:1398] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:03.787477 2499 tablet_bootstrap.cc:492] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:03.787595 2499 ts_tablet_manager.cc:1367] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:03.788769 2499 raft_consensus.cc:348] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.788864 2499 raft_consensus.cc:374] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:03.788892 2499 raft_consensus.cc:729] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:03.788998 2499 consensus_queue.cc:260] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.790005 2499 ts_tablet_manager.cc:1398] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:03.810026 2046 raft_consensus.cc:482] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:03.810142 2046 raft_consensus.cc:504] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.810556 2046 leader_election.cc:288] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:03.810997 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f76c356c24b8aa5b0c4ac92beabc5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:03.811084 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f76c356c24b8aa5b0c4ac92beabc5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:03.811156 24630 raft_consensus.cc:2454] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:03.811247 27577 raft_consensus.cc:2454] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:03.811467 24965 leader_election.cc:302] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:03.811754 2046 raft_consensus.cc:2781] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:03.811825 2046 raft_consensus.cc:482] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:03.811874 2046 raft_consensus.cc:3037] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.812860 2046 raft_consensus.cc:504] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.813181 2046 leader_election.cc:288] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:03.813493 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f76c356c24b8aa5b0c4ac92beabc5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:03.813539 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f76c356c24b8aa5b0c4ac92beabc5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:03.813632 24630 raft_consensus.cc:3037] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.813675 27577 raft_consensus.cc:3037] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:03.814639 24630 raft_consensus.cc:2454] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:03.814639 27577 raft_consensus.cc:2454] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:03.814900 24965 leader_election.cc:302] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:03.815171 2046 raft_consensus.cc:2781] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:03.815268 2046 raft_consensus.cc:686] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:03.815430 2046 consensus_queue.cc:237] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:03.818068 24482 catalog_manager.cc:5484] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:03.893247 2400 consensus_queue.cc:1035] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.898372 2261 consensus_queue.cc:1035] T 896090c872314da4a317cd155f9807bd P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:03.929749 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:03.930965 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:03.935571 27577 raft_consensus.cc:1261] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:03.935576 24630 raft_consensus.cc:1261] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:03.936120 2046 consensus_queue.cc:1035] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.936448 2121 consensus_queue.cc:1035] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:03.939270 24486 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 793f76c356c24b8aa5b0c4ac92beabc5 (table rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b]). Expected version 1 got 0
I20240502 01:59:03.940436 2516 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:03.940639 24486 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 793f76c356c24b8aa5b0c4ac92beabc5 (table rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b]). Expected version 1 got 0
I20240502 01:59:03.940757 2516 tablet.cc:1620] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:03.940897 2432 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:03.941174 2432 tablet.cc:1620] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:03.944414 24481 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 793f76c356c24b8aa5b0c4ac92beabc5 (table rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b]). Expected version 1 got 0
I20240502 01:59:03.945693 2517 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:03.946086 2517 tablet.cc:1620] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:03.947573 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b] alter complete (version 1)
I20240502 01:59:04.040280 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:04.046159 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4be1d96013448dae2746ec28a21536 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.046473 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4be1d96013448dae2746ec28a21536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.046691 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 94983ca5787e4b6cafd92bbf3b8c90b3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.046780 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4be1d96013448dae2746ec28a21536 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.046955 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94983ca5787e4b6cafd92bbf3b8c90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.046952 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 94983ca5787e4b6cafd92bbf3b8c90b3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.047026 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4be1d96013448dae2746ec28a21536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.047194 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94983ca5787e4b6cafd92bbf3b8c90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.049062 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 94983ca5787e4b6cafd92bbf3b8c90b3 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.049180 2498 tablet_bootstrap.cc:492] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:04.049312 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94983ca5787e4b6cafd92bbf3b8c90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.049670 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:04.050104 2498 tablet_bootstrap.cc:654] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.051100 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 830b042d01e1439481f264d4bb08566c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.051333 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830b042d01e1439481f264d4bb08566c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.051800 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 830b042d01e1439481f264d4bb08566c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.051980 2538 tablet_bootstrap.cc:492] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:04.052018 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830b042d01e1439481f264d4bb08566c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:59:04.052069 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:04.052824 2538 tablet_bootstrap.cc:654] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.053751 2538 tablet_bootstrap.cc:492] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:04.053828 2538 ts_tablet_manager.cc:1367] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:04.054248 2538 raft_consensus.cc:348] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.054345 2538 raft_consensus.cc:374] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.054371 2538 raft_consensus.cc:729] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.054479 2538 consensus_queue.cc:260] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.054641 2498 tablet_bootstrap.cc:492] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:04.054720 2498 ts_tablet_manager.cc:1367] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:04.054738 2538 ts_tablet_manager.cc:1398] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:04.054898 2538 tablet_bootstrap.cc:492] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:04.054934 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4be1d96013448dae2746ec28a21536 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.055210 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4be1d96013448dae2746ec28a21536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.055161 2498 raft_consensus.cc:348] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.055262 2498 raft_consensus.cc:374] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.055294 2498 raft_consensus.cc:729] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.055405 2498 consensus_queue.cc:260] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.055685 2498 ts_tablet_manager.cc:1398] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.055878 2538 tablet_bootstrap.cc:654] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.057109 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 830b042d01e1439481f264d4bb08566c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7973a3dc813740a1ad82b3b87110dc0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.057353 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830b042d01e1439481f264d4bb08566c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.057611 2499 tablet_bootstrap.cc:492] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:04.057857 2542 tablet_bootstrap.cc:492] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:04.058492 2499 tablet_bootstrap.cc:654] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.060664 2542 tablet_bootstrap.cc:654] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.061652 2499 tablet_bootstrap.cc:492] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:04.061738 2499 ts_tablet_manager.cc:1367] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:59:04.062141 2546 tablet_bootstrap.cc:492] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:04.062199 2499 raft_consensus.cc:348] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.062295 2499 raft_consensus.cc:374] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.062327 2499 raft_consensus.cc:729] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.062433 2499 consensus_queue.cc:260] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.062705 2499 ts_tablet_manager.cc:1398] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.063385 2546 tablet_bootstrap.cc:654] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.063889 2539 tablet_bootstrap.cc:492] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:04.064646 2538 tablet_bootstrap.cc:492] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:04.064723 2538 ts_tablet_manager.cc:1367] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:59:04.064853 2539 tablet_bootstrap.cc:654] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.065156 2538 raft_consensus.cc:348] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.065248 2538 raft_consensus.cc:374] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.065276 2538 raft_consensus.cc:729] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.065382 2538 consensus_queue.cc:260] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.065670 2538 ts_tablet_manager.cc:1398] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:04.067087 2516 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:04.067382 2516 tablet.cc:1620] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:04.067343 2432 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:04.067579 2432 tablet.cc:1620] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:04.067672 2517 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:04.067917 2517 tablet.cc:1620] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:04.069964 2542 tablet_bootstrap.cc:492] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:04.070056 2542 ts_tablet_manager.cc:1367] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:59:04.070467 2542 raft_consensus.cc:348] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.070572 2542 raft_consensus.cc:374] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.070609 2542 raft_consensus.cc:729] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.070719 2542 consensus_queue.cc:260] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.070986 2542 ts_tablet_manager.cc:1398] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.071156 2542 tablet_bootstrap.cc:492] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:04.072104 2542 tablet_bootstrap.cc:654] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.076427 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=12f3886fff544a56b31a254ed9bf020b] alter complete (version 2)
I20240502 01:59:04.080466 2539 tablet_bootstrap.cc:492] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:04.080540 2539 ts_tablet_manager.cc:1367] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240502 01:59:04.081224 2542 tablet_bootstrap.cc:492] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:04.081187 2539 raft_consensus.cc:348] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.081311 2542 ts_tablet_manager.cc:1367] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 01:59:04.081276 2539 raft_consensus.cc:374] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.081537 2539 raft_consensus.cc:729] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.081656 2539 consensus_queue.cc:260] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.081686 2542 raft_consensus.cc:348] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.081768 2542 raft_consensus.cc:374] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.081805 2542 raft_consensus.cc:729] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.081979 2539 ts_tablet_manager.cc:1398] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.082039 2542 consensus_queue.cc:260] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.082185 2546 tablet_bootstrap.cc:492] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:04.082329 2546 ts_tablet_manager.cc:1367] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 01:59:04.082722 2539 tablet_bootstrap.cc:492] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:04.082749 2546 raft_consensus.cc:348] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.082885 2546 raft_consensus.cc:374] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.082930 2546 raft_consensus.cc:729] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.083246 2546 consensus_queue.cc:260] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.084100 2546 ts_tablet_manager.cc:1398] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:04.085502 2542 ts_tablet_manager.cc:1398] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:59:04.086196 2539 tablet_bootstrap.cc:654] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.089368 2539 tablet_bootstrap.cc:492] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:04.089437 2539 ts_tablet_manager.cc:1367] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:59:04.089762 2539 raft_consensus.cc:348] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.089843 2539 raft_consensus.cc:374] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.089872 2539 raft_consensus.cc:729] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.090076 2539 consensus_queue.cc:260] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.090328 2539 ts_tablet_manager.cc:1398] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.099069 2261 raft_consensus.cc:482] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.099143 2261 raft_consensus.cc:504] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.099555 2261 leader_election.cc:288] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:04.099799 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830b042d01e1439481f264d4bb08566c" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:04.099915 24765 raft_consensus.cc:2454] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:04.099897 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830b042d01e1439481f264d4bb08566c" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:04.099992 27175 raft_consensus.cc:2454] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:04.100191 24833 leader_election.cc:302] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:04.100365 2261 raft_consensus.cc:2781] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.100407 2261 raft_consensus.cc:482] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.100441 2261 raft_consensus.cc:3037] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.101085 2261 raft_consensus.cc:504] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.101389 2261 leader_election.cc:288] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:04.101713 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830b042d01e1439481f264d4bb08566c" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:04.101811 27175 raft_consensus.cc:3037] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.101832 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830b042d01e1439481f264d4bb08566c" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:04.101948 24765 raft_consensus.cc:3037] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.102782 27175 raft_consensus.cc:2454] T 830b042d01e1439481f264d4bb08566c P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:04.102782 24765 raft_consensus.cc:2454] T 830b042d01e1439481f264d4bb08566c P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:04.103040 24833 leader_election.cc:302] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:04.103267 2261 raft_consensus.cc:2781] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.103338 2261 raft_consensus.cc:686] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:04.103461 2261 consensus_queue.cc:237] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.105573 24483 catalog_manager.cc:5484] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:04.147475 2261 raft_consensus.cc:482] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.147600 2261 raft_consensus.cc:504] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.147998 2261 leader_election.cc:288] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:04.148234 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94983ca5787e4b6cafd92bbf3b8c90b3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:04.148357 27444 raft_consensus.cc:2454] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:04.148325 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94983ca5787e4b6cafd92bbf3b8c90b3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:04.148476 24765 raft_consensus.cc:2454] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:04.148641 24833 leader_election.cc:302] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
W20240502 01:59:04.148816 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.148914 2261 raft_consensus.cc:2781] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.148972 2261 raft_consensus.cc:482] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.149011 2261 raft_consensus.cc:3037] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.150110 2261 raft_consensus.cc:504] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.150444 2261 leader_election.cc:288] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:04.150719 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94983ca5787e4b6cafd92bbf3b8c90b3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:04.150787 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94983ca5787e4b6cafd92bbf3b8c90b3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:04.150848 27444 raft_consensus.cc:3037] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.150900 24765 raft_consensus.cc:3037] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.151898 27444 raft_consensus.cc:2454] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:04.151895 24765 raft_consensus.cc:2454] T 94983ca5787e4b6cafd92bbf3b8c90b3 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:04.152171 24833 leader_election.cc:302] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:04.152329 2261 raft_consensus.cc:2781] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.152451 2261 raft_consensus.cc:686] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:04.152586 2261 consensus_queue.cc:237] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:04.154911 24483 catalog_manager.cc:5484] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:04.156813 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:04.166023 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:04.169236 2199 raft_consensus.cc:482] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.169337 2199 raft_consensus.cc:504] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.169817 2199 leader_election.cc:288] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:04.170166 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4be1d96013448dae2746ec28a21536" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:04.170159 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4be1d96013448dae2746ec28a21536" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:04.170295 24632 raft_consensus.cc:2454] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:59:04.170295 27444 raft_consensus.cc:2454] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:59:04.170570 27514 leader_election.cc:302] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:04.171020 2199 raft_consensus.cc:2781] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.171092 2199 raft_consensus.cc:482] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.171160 2199 raft_consensus.cc:3037] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.172420 2199 raft_consensus.cc:504] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.173102 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4be1d96013448dae2746ec28a21536" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:04.173135 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4be1d96013448dae2746ec28a21536" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:04.173209 24632 raft_consensus.cc:3037] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.173245 27444 raft_consensus.cc:3037] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.174259 24632 raft_consensus.cc:2454] T 3d4be1d96013448dae2746ec28a21536 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:59:04.174506 27513 leader_election.cc:302] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:04.174697 2480 raft_consensus.cc:2781] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.174893 2480 raft_consensus.cc:686] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:59:04.175056 2480 consensus_queue.cc:237] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.175987 2199 leader_election.cc:288] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:04.176272 27444 raft_consensus.cc:2454] T 3d4be1d96013448dae2746ec28a21536 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:59:04.177701 24483 catalog_manager.cc:5484] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:04.179339 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:04.190659 2516 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:04.191090 2517 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:04.195720 2432 tablet.cc:1722] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 01:59:04.196990 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:04.197194 24483 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=12f3886fff544a56b31a254ed9bf020b] alter complete (version 3)
W20240502 01:59:04.223650 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:04.302052 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:04.302229 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:04.304383 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 793f76c356c24b8aa5b0c4ac92beabc5
I20240502 01:59:04.305039 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 793f76c356c24b8aa5b0c4ac92beabc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:04 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:04.305179 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 793f76c356c24b8aa5b0c4ac92beabc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:04 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:59:04.305248 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 793f76c356c24b8aa5b0c4ac92beabc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:04 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:59:04.305367 2581 tablet_replica.cc:330] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:04.305533 2581 raft_consensus.cc:2229] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:04.305589 2582 tablet_replica.cc:330] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:59:04.305713 2583 tablet_replica.cc:330] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:59:04.305753 2581 raft_consensus.cc:2258] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:04.305778 2582 raft_consensus.cc:2229] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:04.305853 2583 raft_consensus.cc:2229] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:04.305956 2582 raft_consensus.cc:2258] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:04.306013 2583 raft_consensus.cc:2258] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:04.306367 2581 ts_tablet_manager.cc:1875] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:04.306530 2583 ts_tablet_manager.cc:1875] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:04.306710 2582 ts_tablet_manager.cc:1875] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:04.308588 2583 ts_tablet_manager.cc:1888] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:04.308653 2583 log.cc:1196] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/793f76c356c24b8aa5b0c4ac92beabc5
I20240502 01:59:04.308820 2582 ts_tablet_manager.cc:1888] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:04.308832 2581 ts_tablet_manager.cc:1888] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:04.308907 2581 log.cc:1196] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/793f76c356c24b8aa5b0c4ac92beabc5
I20240502 01:59:04.308923 2583 ts_tablet_manager.cc:1909] T 793f76c356c24b8aa5b0c4ac92beabc5 P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:59:04.308948 2582 log.cc:1196] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/793f76c356c24b8aa5b0c4ac92beabc5
I20240502 01:59:04.309231 2581 ts_tablet_manager.cc:1909] T 793f76c356c24b8aa5b0c4ac92beabc5 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:04.309247 2582 ts_tablet_manager.cc:1909] T 793f76c356c24b8aa5b0c4ac92beabc5 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:59:04.309569 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 793f76c356c24b8aa5b0c4ac92beabc5 (table rebalancer_dynamic_table_new_name [id=12f3886fff544a56b31a254ed9bf020b]) successfully deleted
I20240502 01:59:04.309823 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 793f76c356c24b8aa5b0c4ac92beabc5 (table rebalancer_dynamic_table_new_name [id=12f3886fff544a56b31a254ed9bf020b]) successfully deleted
I20240502 01:59:04.309867 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 793f76c356c24b8aa5b0c4ac92beabc5 (table rebalancer_dynamic_table_new_name [id=12f3886fff544a56b31a254ed9bf020b]) successfully deleted
I20240502 01:59:04.406374 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:04.412024 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]), partition=
I20240502 01:59:04.412079 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]), partition=
I20240502 01:59:04.412220 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]), partition=
I20240502 01:59:04.412307 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f35e9b69a484cca8b1fe733037ca6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.412379 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f35e9b69a484cca8b1fe733037ca6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.412429 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f35e9b69a484cca8b1fe733037ca6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.414572 2546 tablet_bootstrap.cc:492] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:04.414592 2542 tablet_bootstrap.cc:492] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:04.415339 2584 tablet_bootstrap.cc:492] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:04.415477 2542 tablet_bootstrap.cc:654] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.415518 2546 tablet_bootstrap.cc:654] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.416399 2584 tablet_bootstrap.cc:654] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.417399 2542 tablet_bootstrap.cc:492] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:04.417491 2542 ts_tablet_manager.cc:1367] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:04.417881 2546 tablet_bootstrap.cc:492] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:04.417997 2546 ts_tablet_manager.cc:1367] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:04.417945 2542 raft_consensus.cc:348] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
W20240502 01:59:04.418022 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:04.418051 2542 raft_consensus.cc:374] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.418081 2542 raft_consensus.cc:729] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.418215 2542 consensus_queue.cc:260] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.418402 2546 raft_consensus.cc:348] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.418514 2546 raft_consensus.cc:374] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.418506 2542 ts_tablet_manager.cc:1398] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.418535 2546 raft_consensus.cc:729] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.418637 2546 consensus_queue.cc:260] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.419028 2546 ts_tablet_manager.cc:1398] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:04.420491 2584 tablet_bootstrap.cc:492] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:04.420583 2584 ts_tablet_manager.cc:1367] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:04.421160 2584 raft_consensus.cc:348] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.421269 2584 raft_consensus.cc:374] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.421314 2584 raft_consensus.cc:729] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.421445 2584 consensus_queue.cc:260] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.421809 2584 ts_tablet_manager.cc:1398] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:04.451793 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:04.468981 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.487205 2261 consensus_queue.cc:1035] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.491526 1800 consensus_queue.cc:1035] T 830b042d01e1439481f264d4bb08566c P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:04.507917 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.605836 1891 raft_consensus.cc:482] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.605967 1891 raft_consensus.cc:504] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.606333 1891 leader_election.cc:288] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:04.606700 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f35e9b69a484cca8b1fe733037ca6a2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:04.606688 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f35e9b69a484cca8b1fe733037ca6a2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:04.606828 27175 raft_consensus.cc:2454] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:04.606833 24897 raft_consensus.cc:2454] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:04.607113 24966 leader_election.cc:302] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:04.607321 1891 raft_consensus.cc:2781] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.607379 1891 raft_consensus.cc:482] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.607417 1891 raft_consensus.cc:3037] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.608331 1891 raft_consensus.cc:504] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.608668 1891 leader_election.cc:288] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:04.609059 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f35e9b69a484cca8b1fe733037ca6a2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:04.609083 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f35e9b69a484cca8b1fe733037ca6a2" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:04.609162 24897 raft_consensus.cc:3037] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.609210 27175 raft_consensus.cc:3037] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.610201 24897 raft_consensus.cc:2454] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:04.610244 27175 raft_consensus.cc:2454] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:04.610494 24966 leader_election.cc:302] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:04.610690 2121 raft_consensus.cc:2781] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.610814 2121 raft_consensus.cc:686] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:04.610949 2121 consensus_queue.cc:237] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:04.613065 24479 catalog_manager.cc:5484] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:04.641072 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.657698 2400 consensus_queue.cc:1035] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.660895 2261 consensus_queue.cc:1035] T 94983ca5787e4b6cafd92bbf3b8c90b3 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.698401 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:04.704469 27157 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec193ff5a7e4992849ad04d549a2769 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.704509 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec193ff5a7e4992849ad04d549a2769 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.704816 27157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec193ff5a7e4992849ad04d549a2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.704831 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec193ff5a7e4992849ad04d549a2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.704905 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeb5da68f7344978501abc92347e7c9 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.705161 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeb5da68f7344978501abc92347e7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.705135 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec193ff5a7e4992849ad04d549a2769 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.705410 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec193ff5a7e4992849ad04d549a2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.707387 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeb5da68f7344978501abc92347e7c9 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.707404 27558 tablet_service.cc:1467] Processing CreateTablet for tablet f027969435ca41bf8379e9c1ed310cf1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.707633 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f027969435ca41bf8379e9c1ed310cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.707633 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeb5da68f7344978501abc92347e7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.707679 2584 tablet_bootstrap.cc:492] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:04.708184 2546 tablet_bootstrap.cc:492] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:04.708747 2584 tablet_bootstrap.cc:654] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.709172 2546 tablet_bootstrap.cc:654] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.710111 2584 tablet_bootstrap.cc:492] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:04.710206 2584 ts_tablet_manager.cc:1367] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:04.710218 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeb5da68f7344978501abc92347e7c9 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.710487 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeb5da68f7344978501abc92347e7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.710788 2653 tablet_bootstrap.cc:492] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:04.710726 2584 raft_consensus.cc:348] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.710835 2584 raft_consensus.cc:374] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.710867 2584 raft_consensus.cc:729] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.710991 2584 consensus_queue.cc:260] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.711334 2584 ts_tablet_manager.cc:1398] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:04.711910 2653 tablet_bootstrap.cc:654] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.712738 2655 tablet_bootstrap.cc:492] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:04.713900 2655 tablet_bootstrap.cc:654] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.715646 24610 tablet_service.cc:1467] Processing CreateTablet for tablet f027969435ca41bf8379e9c1ed310cf1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.715909 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f027969435ca41bf8379e9c1ed310cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.716045 2655 tablet_bootstrap.cc:492] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:04.716123 2655 ts_tablet_manager.cc:1367] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:04.716594 2655 raft_consensus.cc:348] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.716697 2655 raft_consensus.cc:374] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.716730 2655 raft_consensus.cc:729] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.716869 2655 consensus_queue.cc:260] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.717182 2655 ts_tablet_manager.cc:1398] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:04.718297 25012 tablet_service.cc:1467] Processing CreateTablet for tablet f027969435ca41bf8379e9c1ed310cf1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=08018af7b67a4e80ae80f49ad2262fa7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:04.718538 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f027969435ca41bf8379e9c1ed310cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:04.719763 2651 tablet_bootstrap.cc:492] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
W20240502 01:59:04.719813 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:04.720757 2651 tablet_bootstrap.cc:654] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.721875 2584 tablet_bootstrap.cc:492] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:04.722095 2651 tablet_bootstrap.cc:492] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:04.722168 2651 ts_tablet_manager.cc:1367] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:04.722517 2546 tablet_bootstrap.cc:492] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:04.722594 2546 ts_tablet_manager.cc:1367] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:59:04.722590 2651 raft_consensus.cc:348] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.722679 2651 raft_consensus.cc:374] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.722712 2651 raft_consensus.cc:729] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.722776 2584 tablet_bootstrap.cc:654] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.722812 2651 consensus_queue.cc:260] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.723071 2651 ts_tablet_manager.cc:1398] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:04.723058 2546 raft_consensus.cc:348] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.723147 2546 raft_consensus.cc:374] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.723176 2546 raft_consensus.cc:729] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.723240 2651 tablet_bootstrap.cc:492] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:04.723297 2546 consensus_queue.cc:260] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.723572 2546 ts_tablet_manager.cc:1398] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.723613 2653 tablet_bootstrap.cc:492] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:04.723693 2653 ts_tablet_manager.cc:1367] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:59:04.723731 2546 tablet_bootstrap.cc:492] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:04.724159 2653 raft_consensus.cc:348] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.724261 2651 tablet_bootstrap.cc:654] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.724258 2653 raft_consensus.cc:374] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.724293 2653 raft_consensus.cc:729] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.724411 2653 consensus_queue.cc:260] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.724586 2546 tablet_bootstrap.cc:654] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.724692 2653 ts_tablet_manager.cc:1398] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:04.724841 2653 tablet_bootstrap.cc:492] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:04.725739 2653 tablet_bootstrap.cc:654] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:04.725786 2546 tablet_bootstrap.cc:492] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:04.725853 2546 ts_tablet_manager.cc:1367] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:04.726255 2546 raft_consensus.cc:348] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.726339 2546 raft_consensus.cc:374] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.726365 2546 raft_consensus.cc:729] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.726462 2546 consensus_queue.cc:260] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.726969 2546 ts_tablet_manager.cc:1398] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:04.727569 2651 tablet_bootstrap.cc:492] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:04.727691 2584 tablet_bootstrap.cc:492] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:04.727785 2584 ts_tablet_manager.cc:1367] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:04.727789 2651 ts_tablet_manager.cc:1367] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:04.728252 2584 raft_consensus.cc:348] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.728268 2651 raft_consensus.cc:348] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.728365 2651 raft_consensus.cc:374] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.728353 2584 raft_consensus.cc:374] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.728394 2651 raft_consensus.cc:729] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.728400 2584 raft_consensus.cc:729] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.728539 2584 consensus_queue.cc:260] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.728579 2651 consensus_queue.cc:260] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.728854 2651 ts_tablet_manager.cc:1398] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.728863 2584 ts_tablet_manager.cc:1398] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.731035 2653 tablet_bootstrap.cc:492] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:04.731153 2653 ts_tablet_manager.cc:1367] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:04.731508 2653 raft_consensus.cc:348] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.731600 2653 raft_consensus.cc:374] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:04.731676 2653 raft_consensus.cc:729] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:04.731796 2653 consensus_queue.cc:260] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.732117 2653 ts_tablet_manager.cc:1398] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:04.756124 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:04.760732 27175 raft_consensus.cc:1261] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:04.761158 2121 consensus_queue.cc:1035] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.761322 24896 raft_consensus.cc:1261] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer e0b9e7df4e8543fa9b8e3ffc24330657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:04.761729 2121 consensus_queue.cc:1035] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.763295 2199 consensus_queue.cc:1035] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.764442 24479 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]). Expected version 1 got 0
I20240502 01:59:04.765347 24486 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]). Expected version 1 got 0
I20240502 01:59:04.767405 24482 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]). Expected version 1 got 0
I20240502 01:59:04.768447 2669 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:04.769089 2559 consensus_queue.cc:1035] T 3d4be1d96013448dae2746ec28a21536 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:04.769613 2669 tablet.cc:1620] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:04.770009 2670 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:04.770439 24486 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d]). Expected version 1 got 0
I20240502 01:59:04.770983 2670 tablet.cc:1620] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:04.776749 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d] alter complete (version 1)
I20240502 01:59:04.778627 2667 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:04.779093 2667 tablet.cc:1620] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
W20240502 01:59:04.806284 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.826148 2121 raft_consensus.cc:482] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.826258 2121 raft_consensus.cc:504] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.826614 2121 leader_election.cc:288] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:04.826931 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb5da68f7344978501abc92347e7c9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:04.826968 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb5da68f7344978501abc92347e7c9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:04.827073 27446 raft_consensus.cc:2454] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:04.827092 27175 raft_consensus.cc:2454] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:04.827338 24966 leader_election.cc:302] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:04.827502 2121 raft_consensus.cc:2781] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.827585 2121 raft_consensus.cc:482] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.827616 2121 raft_consensus.cc:3037] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.828671 2121 raft_consensus.cc:504] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.829032 2121 leader_election.cc:288] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:04.829349 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb5da68f7344978501abc92347e7c9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:04.829505 27446 raft_consensus.cc:3037] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.829499 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeb5da68f7344978501abc92347e7c9" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:04.829628 27175 raft_consensus.cc:3037] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.830356 27446 raft_consensus.cc:2454] T 5eeb5da68f7344978501abc92347e7c9 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:04.830452 27175 raft_consensus.cc:2454] T 5eeb5da68f7344978501abc92347e7c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:04.830682 24966 leader_election.cc:302] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:04.830853 2121 raft_consensus.cc:2781] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.830935 2121 raft_consensus.cc:686] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:04.831074 2121 consensus_queue.cc:237] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.833564 24486 catalog_manager.cc:5484] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:04.880404 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:04.886579 2667 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:04.886914 2667 tablet.cc:1620] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:04.887631 2670 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:04.887929 2670 tablet.cc:1620] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:04.893736 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=1c8bb93dc48c4df487e82a3c5d8f413d] alter complete (version 2)
I20240502 01:59:04.896006 2669 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:04.896289 2669 tablet.cc:1620] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:04.929545 2389 raft_consensus.cc:482] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.929653 2389 raft_consensus.cc:504] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.930035 2389 leader_election.cc:288] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:04.930318 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f027969435ca41bf8379e9c1ed310cf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:04.930450 27577 raft_consensus.cc:2454] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:04.930760 24567 leader_election.cc:302] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:04.930827 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f027969435ca41bf8379e9c1ed310cf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:04.930955 2389 raft_consensus.cc:2781] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.930958 25032 raft_consensus.cc:2454] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:04.931013 2389 raft_consensus.cc:482] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.931046 2389 raft_consensus.cc:3037] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.932257 2389 raft_consensus.cc:504] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.932579 2389 leader_election.cc:288] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:04.932986 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f027969435ca41bf8379e9c1ed310cf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:04.933099 27577 raft_consensus.cc:3037] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.933852 27577 raft_consensus.cc:2454] T f027969435ca41bf8379e9c1ed310cf1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:04.934121 24567 leader_election.cc:302] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:04.934257 2389 raft_consensus.cc:2781] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.934329 2389 raft_consensus.cc:686] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:04.934440 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f027969435ca41bf8379e9c1ed310cf1" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:04.934454 2389 consensus_queue.cc:237] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:04.934543 25032 raft_consensus.cc:3037] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.935505 25032 raft_consensus.cc:2454] T f027969435ca41bf8379e9c1ed310cf1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:04.937438 24486 catalog_manager.cc:5484] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:04.944537 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:04.951836 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:04.961805 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.965173 2630 raft_consensus.cc:482] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:04.965272 2630 raft_consensus.cc:504] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.965690 2630 leader_election.cc:288] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:04.966025 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec193ff5a7e4992849ad04d549a2769" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:04.966032 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec193ff5a7e4992849ad04d549a2769" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:04.966156 27577 raft_consensus.cc:2454] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:04.966173 27175 raft_consensus.cc:2454] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:04.966398 24567 leader_election.cc:302] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:04.966559 2630 raft_consensus.cc:2781] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:04.966617 2630 raft_consensus.cc:482] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:04.966660 2630 raft_consensus.cc:3037] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.967525 2630 raft_consensus.cc:504] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.967808 2630 leader_election.cc:288] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:04.968111 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec193ff5a7e4992849ad04d549a2769" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:04.968221 27577 raft_consensus.cc:3037] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.968183 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec193ff5a7e4992849ad04d549a2769" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:04.968305 27175 raft_consensus.cc:3037] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:04.969036 27577 raft_consensus.cc:2454] T 2ec193ff5a7e4992849ad04d549a2769 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:04.969192 27175 raft_consensus.cc:2454] T 2ec193ff5a7e4992849ad04d549a2769 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:04.969431 24566 leader_election.cc:302] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:04.969648 2389 raft_consensus.cc:2781] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:04.969733 2389 raft_consensus.cc:686] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:04.969864 2389 consensus_queue.cc:237] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:04.971870 24486 catalog_manager.cc:5484] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:04.983633 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:04.989552 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:04.995784 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:05.001106 2667 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:05.001583 2670 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:05.004488 2669 tablet.cc:1722] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:05.006651 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=1c8bb93dc48c4df487e82a3c5d8f413d] alter complete (version 3)
W20240502 01:59:05.032490 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:59:05.033131 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:05.078284 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 01:59:05.108649 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:05.108836 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240502 01:59:05.110782 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.111265 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 2f35e9b69a484cca8b1fe733037ca6a2
I20240502 01:59:05.111737 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f35e9b69a484cca8b1fe733037ca6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:05 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:59:05.111871 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f35e9b69a484cca8b1fe733037ca6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:05 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:05.112128 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f35e9b69a484cca8b1fe733037ca6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:05 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:59:05.112165 2701 tablet_replica.cc:330] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:59:05.112329 2701 raft_consensus.cc:2229] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:05.112476 2701 raft_consensus.cc:2258] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:05.112505 2704 tablet_replica.cc:330] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:59:05.112684 2704 raft_consensus.cc:2229] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:05.112825 2702 tablet_replica.cc:330] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:05.112841 2704 raft_consensus.cc:2258] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:05.113185 2701 ts_tablet_manager.cc:1875] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:05.113200 2702 raft_consensus.cc:2229] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:05.113369 2704 ts_tablet_manager.cc:1875] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:05.113405 2702 raft_consensus.cc:2258] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:05.113996 2702 ts_tablet_manager.cc:1875] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:05.115375 2701 ts_tablet_manager.cc:1888] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:05.115481 2701 log.cc:1196] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f35e9b69a484cca8b1fe733037ca6a2
W20240502 01:59:05.115736 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 01:59:05.115916 2704 ts_tablet_manager.cc:1888] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:05.116005 2704 log.cc:1196] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/2f35e9b69a484cca8b1fe733037ca6a2
I20240502 01:59:05.116045 2701 ts_tablet_manager.cc:1909] T 2f35e9b69a484cca8b1fe733037ca6a2 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:59:05.116272 2704 ts_tablet_manager.cc:1909] T 2f35e9b69a484cca8b1fe733037ca6a2 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:59:05.116509 2702 ts_tablet_manager.cc:1888] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:05.116590 2702 log.cc:1196] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/2f35e9b69a484cca8b1fe733037ca6a2
I20240502 01:59:05.116871 2702 ts_tablet_manager.cc:1909] T 2f35e9b69a484cca8b1fe733037ca6a2 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:05.116835 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table_new_name [id=1c8bb93dc48c4df487e82a3c5d8f413d]) successfully deleted
I20240502 01:59:05.117588 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table_new_name [id=1c8bb93dc48c4df487e82a3c5d8f413d]) successfully deleted
I20240502 01:59:05.117898 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 2f35e9b69a484cca8b1fe733037ca6a2 (table rebalancer_dynamic_table_new_name [id=1c8bb93dc48c4df487e82a3c5d8f413d]) successfully deleted
W20240502 01:59:05.163293 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:05.202090 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 01:59:05.202831 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:05.213650 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:05.218976 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 93d2fbfd12044bd79182e8e143ab0335 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]), partition=
I20240502 01:59:05.219071 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 93d2fbfd12044bd79182e8e143ab0335 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]), partition=
I20240502 01:59:05.219252 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d2fbfd12044bd79182e8e143ab0335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.219301 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d2fbfd12044bd79182e8e143ab0335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.221310 2651 tablet_bootstrap.cc:492] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:05.221570 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 93d2fbfd12044bd79182e8e143ab0335 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]), partition=
I20240502 01:59:05.221612 2731 tablet_bootstrap.cc:492] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:05.221797 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d2fbfd12044bd79182e8e143ab0335. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.222280 2651 tablet_bootstrap.cc:654] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.222513 2731 tablet_bootstrap.cc:654] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.223381 2651 tablet_bootstrap.cc:492] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:05.223466 2651 ts_tablet_manager.cc:1367] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:05.223816 2731 tablet_bootstrap.cc:492] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:05.223896 2731 ts_tablet_manager.cc:1367] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:05.223943 2651 raft_consensus.cc:348] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.224049 2651 raft_consensus.cc:374] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.224082 2651 raft_consensus.cc:729] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.224186 2651 consensus_queue.cc:260] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.224458 2651 ts_tablet_manager.cc:1398] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.224540 2734 tablet_bootstrap.cc:492] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:05.224918 2731 raft_consensus.cc:348] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.225034 2731 raft_consensus.cc:374] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.225090 2731 raft_consensus.cc:729] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.225226 2731 consensus_queue.cc:260] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.225796 2734 tablet_bootstrap.cc:654] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.227413 1891 consensus_queue.cc:1035] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:05.227857 2731 ts_tablet_manager.cc:1398] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240502 01:59:05.228327 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.228595 2734 tablet_bootstrap.cc:492] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:05.228724 2734 ts_tablet_manager.cc:1367] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:59:05.229357 2734 raft_consensus.cc:348] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.229468 2734 raft_consensus.cc:374] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.229496 2734 raft_consensus.cc:729] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.229609 2734 consensus_queue.cc:260] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.229899 2734 ts_tablet_manager.cc:1398] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:05.234552 2515 consensus_queue.cc:1035] T 5eeb5da68f7344978501abc92347e7c9 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:05.280336 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:05.301568 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:05.345402 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:05.359154 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:05.379437 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.394691 2228 raft_consensus.cc:482] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:05.394802 2228 raft_consensus.cc:504] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.395242 2228 leader_election.cc:288] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:05.395522 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d2fbfd12044bd79182e8e143ab0335" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:05.395671 24765 raft_consensus.cc:2454] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:05.395675 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d2fbfd12044bd79182e8e143ab0335" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:05.395804 24632 raft_consensus.cc:2454] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:05.396282 27379 leader_election.cc:302] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:59:05.396457 2228 raft_consensus.cc:2781] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:05.396519 2228 raft_consensus.cc:482] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:05.396556 2228 raft_consensus.cc:3037] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.398331 2228 raft_consensus.cc:504] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.398670 2228 leader_election.cc:288] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:05.399144 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d2fbfd12044bd79182e8e143ab0335" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:05.399233 24632 raft_consensus.cc:3037] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.399565 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d2fbfd12044bd79182e8e143ab0335" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:05.399691 24765 raft_consensus.cc:3037] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.400036 24632 raft_consensus.cc:2454] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:05.400382 27379 leader_election.cc:302] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:59:05.400545 2228 raft_consensus.cc:2781] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:05.400612 2228 raft_consensus.cc:686] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:05.400725 24765 raft_consensus.cc:2454] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:05.400728 2228 consensus_queue.cc:237] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:05.403704 24481 catalog_manager.cc:5484] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:05.449218 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.457593 2389 consensus_queue.cc:1035] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:05.461282 2389 consensus_queue.cc:1035] T 2ec193ff5a7e4992849ad04d549a2769 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:05.479933 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 01:59:05.510170 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:05.514915 24632 raft_consensus.cc:1261] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:05.515096 24765 raft_consensus.cc:1261] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:59:05.515137 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.515470 2228 consensus_queue.cc:1035] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:05.515662 2730 consensus_queue.cc:1035] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:05.518528 24479 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]). Expected version 1 got 0
I20240502 01:59:05.519805 2773 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:05.520016 24481 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]). Expected version 1 got 0
I20240502 01:59:05.520165 2773 tablet.cc:1620] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:05.521333 24479 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]). Expected version 1 got 0
I20240502 01:59:05.521971 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:05.522804 24486 catalog_manager.cc:5576] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237) does not have the latest schema for tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]). Expected version 1 got 0
I20240502 01:59:05.527722 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29] alter complete (version 1)
I20240502 01:59:05.528113 2774 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:05.528404 2774 tablet.cc:1620] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:05.528949 2776 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:05.529235 2776 tablet.cc:1620] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:59:05.529395 24486 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29]). Expected version 1 got 0
I20240502 01:59:05.530109 27156 tablet_service.cc:1467] Processing CreateTablet for tablet ab2d54fd9bcc4d7e80731fd03b23feb9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.530208 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 8066481807d44a1db5eb6a09cc97b3d3 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.530367 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2d54fd9bcc4d7e80731fd03b23feb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.530496 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8066481807d44a1db5eb6a09cc97b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.530696 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 5b541882bb434d5bba72511931d03ba8 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.530704 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 8066481807d44a1db5eb6a09cc97b3d3 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.530948 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b541882bb434d5bba72511931d03ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.531082 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8066481807d44a1db5eb6a09cc97b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.532812 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 5b541882bb434d5bba72511931d03ba8 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.533087 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b541882bb434d5bba72511931d03ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.533087 27557 tablet_service.cc:1467] Processing CreateTablet for tablet ab2d54fd9bcc4d7e80731fd03b23feb9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.533303 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2d54fd9bcc4d7e80731fd03b23feb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.533345 2780 tablet_bootstrap.cc:492] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:05.534456 2780 tablet_bootstrap.cc:654] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.535441 24610 tablet_service.cc:1467] Processing CreateTablet for tablet ab2d54fd9bcc4d7e80731fd03b23feb9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.535697 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2d54fd9bcc4d7e80731fd03b23feb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.535851 2731 tablet_bootstrap.cc:492] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:05.536461 2779 tablet_bootstrap.cc:492] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:05.536899 2731 tablet_bootstrap.cc:654] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.537421 2651 tablet_bootstrap.cc:492] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:05.537353 2779 tablet_bootstrap.cc:654] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.537963 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 5b541882bb434d5bba72511931d03ba8 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.538204 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b541882bb434d5bba72511931d03ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.538331 2651 tablet_bootstrap.cc:654] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.538733 2779 tablet_bootstrap.cc:492] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:05.538812 2779 ts_tablet_manager.cc:1367] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:05.539297 2779 raft_consensus.cc:348] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.539408 2779 raft_consensus.cc:374] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.539445 2731 tablet_bootstrap.cc:492] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:05.539449 2779 raft_consensus.cc:729] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.539522 2731 ts_tablet_manager.cc:1367] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:05.539569 2779 consensus_queue.cc:260] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.539847 2779 ts_tablet_manager.cc:1398] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.539930 2731 raft_consensus.cc:348] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.540026 2731 raft_consensus.cc:374] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.540055 2731 raft_consensus.cc:729] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.540179 2731 consensus_queue.cc:260] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.540385 2734 tablet_bootstrap.cc:492] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:05.540467 2731 ts_tablet_manager.cc:1398] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.541319 2734 tablet_bootstrap.cc:654] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.543139 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 8066481807d44a1db5eb6a09cc97b3d3 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=41682811f15543f1a13abfd85d34bcfb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:05.543429 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8066481807d44a1db5eb6a09cc97b3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.545332 2778 tablet_bootstrap.cc:492] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:05.545584 2389 consensus_queue.cc:1035] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:05.546420 2778 tablet_bootstrap.cc:654] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.546948 2767 consensus_queue.cc:1035] T f027969435ca41bf8379e9c1ed310cf1 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:05.547444 2734 tablet_bootstrap.cc:492] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:05.547528 2734 ts_tablet_manager.cc:1367] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:59:05.547789 2651 tablet_bootstrap.cc:492] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:05.547863 2651 ts_tablet_manager.cc:1367] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:59:05.547941 2734 raft_consensus.cc:348] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.548030 2734 raft_consensus.cc:374] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.548056 2734 raft_consensus.cc:729] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.548161 2734 consensus_queue.cc:260] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.548318 2651 raft_consensus.cc:348] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.548403 2651 raft_consensus.cc:374] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.548430 2651 raft_consensus.cc:729] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.548444 2734 ts_tablet_manager.cc:1398] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:05.548552 2651 consensus_queue.cc:260] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.548817 2651 ts_tablet_manager.cc:1398] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:05.550180 2778 tablet_bootstrap.cc:492] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:05.550315 2778 ts_tablet_manager.cc:1367] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:05.550531 2780 tablet_bootstrap.cc:492] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:05.550613 2780 ts_tablet_manager.cc:1367] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:59:05.550786 2778 raft_consensus.cc:348] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.550877 2778 raft_consensus.cc:374] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.550918 2778 raft_consensus.cc:729] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.551028 2778 consensus_queue.cc:260] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.551030 2780 raft_consensus.cc:348] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.551112 2780 raft_consensus.cc:374] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.551162 2780 raft_consensus.cc:729] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.551267 2780 consensus_queue.cc:260] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.551380 2778 ts_tablet_manager.cc:1398] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.551534 2780 ts_tablet_manager.cc:1398] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:05.551680 2780 tablet_bootstrap.cc:492] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:05.552660 2780 tablet_bootstrap.cc:654] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.556028 2780 tablet_bootstrap.cc:492] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:05.556154 2780 ts_tablet_manager.cc:1367] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:05.556614 2780 raft_consensus.cc:348] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.556761 2780 raft_consensus.cc:374] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.556838 2780 raft_consensus.cc:729] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.557004 2780 consensus_queue.cc:260] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.557375 2780 ts_tablet_manager.cc:1398] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:05.557592 2780 tablet_bootstrap.cc:492] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:05.558457 2780 tablet_bootstrap.cc:654] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.559468 2780 tablet_bootstrap.cc:492] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:05.559542 2780 ts_tablet_manager.cc:1367] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:05.559859 2780 raft_consensus.cc:348] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.559954 2780 raft_consensus.cc:374] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.559991 2780 raft_consensus.cc:729] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.560096 2780 consensus_queue.cc:260] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.560326 2780 ts_tablet_manager.cc:1398] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:05.561669 2787 tablet_bootstrap.cc:492] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:05.562695 2787 tablet_bootstrap.cc:654] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.565656 2787 tablet_bootstrap.cc:492] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:05.565748 2787 ts_tablet_manager.cc:1367] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:05.566141 2787 raft_consensus.cc:348] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.566226 2787 raft_consensus.cc:374] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.566246 2787 raft_consensus.cc:729] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.566332 2787 consensus_queue.cc:260] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.566566 2787 ts_tablet_manager.cc:1398] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:05.579690 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:05.580363 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.583791 2630 raft_consensus.cc:482] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:05.583882 2630 raft_consensus.cc:504] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.584210 2630 leader_election.cc:288] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:05.584467 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d54fd9bcc4d7e80731fd03b23feb9" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:05.584484 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d54fd9bcc4d7e80731fd03b23feb9" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:05.584579 27577 raft_consensus.cc:2454] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:05.584605 27177 raft_consensus.cc:2454] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:05.584846 24566 leader_election.cc:302] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:05.585191 2630 raft_consensus.cc:2781] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:05.585250 2630 raft_consensus.cc:482] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:05.585299 2630 raft_consensus.cc:3037] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.586267 2630 raft_consensus.cc:504] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.586587 2630 leader_election.cc:288] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:05.586781 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d54fd9bcc4d7e80731fd03b23feb9" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:05.586892 27577 raft_consensus.cc:3037] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.587826 27577 raft_consensus.cc:2454] T ab2d54fd9bcc4d7e80731fd03b23feb9 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:05.588148 24567 leader_election.cc:302] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:05.588562 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d54fd9bcc4d7e80731fd03b23feb9" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:05.588677 27177 raft_consensus.cc:3037] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.589277 2630 raft_consensus.cc:2781] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:05.589365 2630 raft_consensus.cc:686] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:05.589499 2630 consensus_queue.cc:237] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.589649 27177 raft_consensus.cc:2454] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:05.591890 24486 catalog_manager.cc:5484] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:05.622777 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.641120 2559 raft_consensus.cc:482] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:05.641220 2559 raft_consensus.cc:504] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.641674 2559 leader_election.cc:288] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:05.641990 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b541882bb434d5bba72511931d03ba8" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:05.642066 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b541882bb434d5bba72511931d03ba8" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:05.642133 24763 raft_consensus.cc:2454] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:59:05.642194 27446 raft_consensus.cc:2454] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:59:05.642400 27514 leader_election.cc:302] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:05.642561 2559 raft_consensus.cc:2781] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:05.642618 2559 raft_consensus.cc:482] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:05.642666 2559 raft_consensus.cc:3037] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.643712 2559 raft_consensus.cc:504] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.644212 2559 leader_election.cc:288] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:05.644467 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b541882bb434d5bba72511931d03ba8" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:05.644585 24763 raft_consensus.cc:3037] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.644670 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b541882bb434d5bba72511931d03ba8" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:05.644778 27444 raft_consensus.cc:3037] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.644793 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:05.645525 27444 raft_consensus.cc:2454] T 5b541882bb434d5bba72511931d03ba8 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:59:05.645680 24763 raft_consensus.cc:2454] T 5b541882bb434d5bba72511931d03ba8 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:59:05.646149 27514 leader_election.cc:302] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:05.646363 2559 raft_consensus.cc:2781] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:05.646440 2559 raft_consensus.cc:686] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:59:05.646565 2559 consensus_queue.cc:237] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:05.650084 24486 catalog_manager.cc:5484] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:05.652455 2773 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:05.652796 2773 tablet.cc:1620] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:05.653578 2776 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:05.653877 2776 tablet.cc:1620] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: MemRowSet was empty: no flush needed.
I20240502 01:59:05.653952 2774 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:05.654268 2774 tablet.cc:1620] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:05.660334 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=d7d98f7a6d564c2d8555e88541e0bd29] alter complete (version 2)
W20240502 01:59:05.690681 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:05.732143 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:05.749687 2400 raft_consensus.cc:482] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:05.749809 2400 raft_consensus.cc:504] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.751539 2400 leader_election.cc:288] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:05.751832 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8066481807d44a1db5eb6a09cc97b3d3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:05.751832 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8066481807d44a1db5eb6a09cc97b3d3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:05.751962 27577 raft_consensus.cc:2454] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:05.751971 25032 raft_consensus.cc:2454] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:05.752226 24833 leader_election.cc:302] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:05.752621 2400 raft_consensus.cc:2781] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:05.752683 2400 raft_consensus.cc:482] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:05.752719 2400 raft_consensus.cc:3037] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.753778 2400 raft_consensus.cc:504] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.754155 2400 leader_election.cc:288] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:05.754494 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8066481807d44a1db5eb6a09cc97b3d3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:05.754487 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8066481807d44a1db5eb6a09cc97b3d3" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:05.754602 27577 raft_consensus.cc:3037] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.754602 25032 raft_consensus.cc:3037] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:05.755605 27577 raft_consensus.cc:2454] T 8066481807d44a1db5eb6a09cc97b3d3 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:05.755640 25032 raft_consensus.cc:2454] T 8066481807d44a1db5eb6a09cc97b3d3 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:05.755872 24830 leader_election.cc:302] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:05.756266 2400 raft_consensus.cc:2781] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:05.756526 2400 raft_consensus.cc:686] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:05.756662 2400 consensus_queue.cc:237] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:05.758970 24486 catalog_manager.cc:5484] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:05.762706 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:05.768235 2773 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:05.768810 2776 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:05.774024 2774 tablet.cc:1722] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:05.774329 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=d7d98f7a6d564c2d8555e88541e0bd29] alter complete (version 3)
W20240502 01:59:05.827698 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:05.878598 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:05.878759 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:05.881002 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 93d2fbfd12044bd79182e8e143ab0335
I20240502 01:59:05.881791 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d2fbfd12044bd79182e8e143ab0335 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:05 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:59:05.882020 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d2fbfd12044bd79182e8e143ab0335 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:05 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:59:05.881867 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d2fbfd12044bd79182e8e143ab0335 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:05 UTC) from {username='slave'} at 127.0.0.1:36636
I20240502 01:59:05.882364 2854 tablet_replica.cc:330] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:59:05.882630 2854 raft_consensus.cc:2229] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:05.882838 2854 raft_consensus.cc:2258] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:05.883078 2856 tablet_replica.cc:330] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: stopping tablet replica
I20240502 01:59:05.883301 2856 raft_consensus.cc:2229] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:05.883524 2856 raft_consensus.cc:2258] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:05.883963 2854 ts_tablet_manager.cc:1875] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:05.884285 2856 ts_tablet_manager.cc:1875] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:05.884886 2855 tablet_replica.cc:330] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:59:05.885066 2855 raft_consensus.cc:2229] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:05.885262 2855 raft_consensus.cc:2258] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:05.886317 2855 ts_tablet_manager.cc:1875] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:05.887528 2856 ts_tablet_manager.cc:1888] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:05.887601 2856 log.cc:1196] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-1/wal/wals/93d2fbfd12044bd79182e8e143ab0335
I20240502 01:59:05.887954 2856 ts_tablet_manager.cc:1909] T 93d2fbfd12044bd79182e8e143ab0335 P 92cb785c165749eca49af5784756461c: Deleting consensus metadata
I20240502 01:59:05.888746 24474 catalog_manager.cc:4830] TS 92cb785c165749eca49af5784756461c (127.17.243.2:45237): tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table_new_name [id=d7d98f7a6d564c2d8555e88541e0bd29]) successfully deleted
I20240502 01:59:05.888945 2855 ts_tablet_manager.cc:1888] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:05.889020 2855 log.cc:1196] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/93d2fbfd12044bd79182e8e143ab0335
I20240502 01:59:05.889271 2855 ts_tablet_manager.cc:1909] T 93d2fbfd12044bd79182e8e143ab0335 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:59:05.890139 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table_new_name [id=d7d98f7a6d564c2d8555e88541e0bd29]) successfully deleted
I20240502 01:59:05.890951 2854 ts_tablet_manager.cc:1888] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:05.891049 2854 log.cc:1196] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/93d2fbfd12044bd79182e8e143ab0335
I20240502 01:59:05.891372 2854 ts_tablet_manager.cc:1909] T 93d2fbfd12044bd79182e8e143ab0335 P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:59:05.892186 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 93d2fbfd12044bd79182e8e143ab0335 (table rebalancer_dynamic_table_new_name [id=d7d98f7a6d564c2d8555e88541e0bd29]) successfully deleted
I20240502 01:59:05.983325 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:05.989075 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 4990271fc1144f5da2539a78057e2143 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]), partition=
I20240502 01:59:05.989130 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 4990271fc1144f5da2539a78057e2143 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]), partition=
I20240502 01:59:05.989342 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4990271fc1144f5da2539a78057e2143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.989342 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4990271fc1144f5da2539a78057e2143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.989420 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 4990271fc1144f5da2539a78057e2143 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]), partition=
I20240502 01:59:05.989667 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4990271fc1144f5da2539a78057e2143. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:05.991281 2778 tablet_bootstrap.cc:492] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:05.991539 2779 tablet_bootstrap.cc:492] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:05.991590 2787 tablet_bootstrap.cc:492] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:05.992130 2778 tablet_bootstrap.cc:654] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.992334 2787 tablet_bootstrap.cc:654] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.992367 2779 tablet_bootstrap.cc:654] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:05.993502 2778 tablet_bootstrap.cc:492] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:05.993589 2778 ts_tablet_manager.cc:1367] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:05.994052 2778 raft_consensus.cc:348] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.994150 2778 raft_consensus.cc:374] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.994181 2778 raft_consensus.cc:729] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.994195 2779 tablet_bootstrap.cc:492] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:05.994277 2779 ts_tablet_manager.cc:1367] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:05.994313 2778 consensus_queue.cc:260] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.994567 2778 ts_tablet_manager.cc:1398] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:05.994668 2779 raft_consensus.cc:348] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.994765 2779 raft_consensus.cc:374] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.994792 2779 raft_consensus.cc:729] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.994899 2779 consensus_queue.cc:260] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.995170 2779 ts_tablet_manager.cc:1398] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:05.997073 2787 tablet_bootstrap.cc:492] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:05.997174 2787 ts_tablet_manager.cc:1367] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:05.997586 2787 raft_consensus.cc:348] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.997675 2787 raft_consensus.cc:374] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:05.997702 2787 raft_consensus.cc:729] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:05.997807 2787 consensus_queue.cc:260] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:05.998106 2787 ts_tablet_manager.cc:1398] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:06.021178 2767 consensus_queue.cc:1035] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.024000 2767 consensus_queue.cc:1035] T ab2d54fd9bcc4d7e80731fd03b23feb9 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.076807 2738 consensus_queue.cc:1035] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.080884 2559 consensus_queue.cc:1035] T 5b541882bb434d5bba72511931d03ba8 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.082855 2261 raft_consensus.cc:482] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:06.082960 2261 raft_consensus.cc:504] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.083312 2261 leader_election.cc:288] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:06.083966 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4990271fc1144f5da2539a78057e2143" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:06.084095 27177 raft_consensus.cc:2454] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:06.084764 24832 leader_election.cc:302] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:06.085072 2400 raft_consensus.cc:2781] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:06.085139 2400 raft_consensus.cc:482] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:06.085183 2400 raft_consensus.cc:3037] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.086321 2400 raft_consensus.cc:504] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.086633 2400 leader_election.cc:288] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:06.086808 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4990271fc1144f5da2539a78057e2143" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:06.086953 25032 raft_consensus.cc:2454] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:06.087311 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4990271fc1144f5da2539a78057e2143" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:06.087405 25032 raft_consensus.cc:3037] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.087494 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4990271fc1144f5da2539a78057e2143" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:06.087610 27177 raft_consensus.cc:3037] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.088369 25032 raft_consensus.cc:2454] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:06.088572 27177 raft_consensus.cc:2454] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:06.088668 24830 leader_election.cc:302] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:06.089919 2400 raft_consensus.cc:2781] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:06.090029 2400 raft_consensus.cc:686] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:06.090160 2400 consensus_queue.cc:237] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.096004 24486 catalog_manager.cc:5484] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:06.129487 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:06.180936 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:06.219630 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:06.223723 27177 raft_consensus.cc:1261] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:06.223801 25032 raft_consensus.cc:1261] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:06.224200 2400 consensus_queue.cc:1035] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.224368 2261 consensus_queue.cc:1035] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.224887 2400 consensus_queue.cc:1035] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.227589 24482 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]). Expected version 1 got 0
I20240502 01:59:06.228327 24486 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]). Expected version 1 got 0
I20240502 01:59:06.229632 2899 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:06.230044 2899 tablet.cc:1620] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:06.231021 2900 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:06.231338 2900 tablet.cc:1620] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:06.231974 2400 consensus_queue.cc:1035] T 8066481807d44a1db5eb6a09cc97b3d3 P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.233407 24481 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]). Expected version 1 got 0
I20240502 01:59:06.239322 2903 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 01:59:06.239706 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:06.239722 2903 tablet.cc:1620] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:06.242240 24482 catalog_manager.cc:5576] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239) does not have the latest schema for tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c]). Expected version 1 got 0
I20240502 01:59:06.242404 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c] alter complete (version 1)
I20240502 01:59:06.283833 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:06.290828 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 18207e699e08468c96fe5d99bc20e3c9 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.291195 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18207e699e08468c96fe5d99bc20e3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.291112 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 18207e699e08468c96fe5d99bc20e3c9 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.291451 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18207e699e08468c96fe5d99bc20e3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.292343 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 18207e699e08468c96fe5d99bc20e3c9 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.292632 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18207e699e08468c96fe5d99bc20e3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.293617 24876 tablet_service.cc:1467] Processing CreateTablet for tablet eae4216a686d494a988a6bce810f5eea (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.293853 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae4216a686d494a988a6bce810f5eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.294025 24744 tablet_service.cc:1467] Processing CreateTablet for tablet eae4216a686d494a988a6bce810f5eea (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.294296 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae4216a686d494a988a6bce810f5eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.295982 25012 tablet_service.cc:1467] Processing CreateTablet for tablet fea3d358fc7a4baea6dc66ce8373d0aa (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.296227 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea3d358fc7a4baea6dc66ce8373d0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.296730 2778 tablet_bootstrap.cc:492] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:06.296864 2906 tablet_bootstrap.cc:492] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:06.297782 2906 tablet_bootstrap.cc:654] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.297843 2778 tablet_bootstrap.cc:654] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.298717 27423 tablet_service.cc:1467] Processing CreateTablet for tablet fea3d358fc7a4baea6dc66ce8373d0aa (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.299014 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea3d358fc7a4baea6dc66ce8373d0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.299029 2778 tablet_bootstrap.cc:492] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:06.299111 2778 ts_tablet_manager.cc:1367] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:06.299610 2778 raft_consensus.cc:348] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.299716 2778 raft_consensus.cc:374] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.299746 2778 raft_consensus.cc:729] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.299857 2778 consensus_queue.cc:260] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.300196 2778 ts_tablet_manager.cc:1398] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:06.300940 2787 tablet_bootstrap.cc:492] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:06.301893 2787 tablet_bootstrap.cc:654] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.303365 2787 tablet_bootstrap.cc:492] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:06.303372 24610 tablet_service.cc:1467] Processing CreateTablet for tablet fea3d358fc7a4baea6dc66ce8373d0aa (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.303463 2787 ts_tablet_manager.cc:1367] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:06.303614 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea3d358fc7a4baea6dc66ce8373d0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.303879 2787 raft_consensus.cc:348] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.303973 2787 raft_consensus.cc:374] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.304001 2787 raft_consensus.cc:729] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.304102 2787 consensus_queue.cc:260] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.304389 2787 ts_tablet_manager.cc:1398] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:06.307950 27156 tablet_service.cc:1467] Processing CreateTablet for tablet eae4216a686d494a988a6bce810f5eea (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=6e6d32f9e8794e2682357b2e644e0ce3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:06.308207 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae4216a686d494a988a6bce810f5eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.308967 2910 tablet_bootstrap.cc:492] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:06.310101 2910 tablet_bootstrap.cc:654] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.310549 2779 tablet_bootstrap.cc:492] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:06.310865 2778 tablet_bootstrap.cc:492] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:06.311540 2779 tablet_bootstrap.cc:654] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.311687 2906 tablet_bootstrap.cc:492] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:06.311749 2778 tablet_bootstrap.cc:654] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.311780 2906 ts_tablet_manager.cc:1367] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:59:06.312520 2906 raft_consensus.cc:348] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.312690 2910 tablet_bootstrap.cc:492] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:06.312719 2906 raft_consensus.cc:374] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.312775 2906 raft_consensus.cc:729] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.312901 2910 ts_tablet_manager.cc:1367] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:59:06.312906 2906 consensus_queue.cc:260] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.313266 2906 ts_tablet_manager.cc:1398] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:06.313498 2910 raft_consensus.cc:348] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.313648 2910 raft_consensus.cc:374] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.313726 2910 raft_consensus.cc:729] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.313863 2910 consensus_queue.cc:260] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.314177 2910 ts_tablet_manager.cc:1398] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:06.315104 2778 tablet_bootstrap.cc:492] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:06.315187 2778 ts_tablet_manager.cc:1367] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:06.315603 2778 raft_consensus.cc:348] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.315698 2778 raft_consensus.cc:374] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.315726 2778 raft_consensus.cc:729] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.315833 2778 consensus_queue.cc:260] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.315874 2779 tablet_bootstrap.cc:492] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:06.315955 2779 ts_tablet_manager.cc:1367] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:59:06.316112 2778 ts_tablet_manager.cc:1398] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:06.316473 2779 raft_consensus.cc:348] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.316593 2779 raft_consensus.cc:374] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.316625 2779 raft_consensus.cc:729] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.316736 2779 consensus_queue.cc:260] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.317001 2779 ts_tablet_manager.cc:1398] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:06.317193 2779 tablet_bootstrap.cc:492] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:06.322196 2779 tablet_bootstrap.cc:654] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.323088 2904 tablet_bootstrap.cc:492] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:06.324434 2904 tablet_bootstrap.cc:654] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.325688 2779 tablet_bootstrap.cc:492] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:06.325788 2779 ts_tablet_manager.cc:1367] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:06.325815 2904 tablet_bootstrap.cc:492] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:06.325887 2904 ts_tablet_manager.cc:1367] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:06.326282 2779 raft_consensus.cc:348] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.326316 2904 raft_consensus.cc:348] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.326395 2779 raft_consensus.cc:374] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.326413 2904 raft_consensus.cc:374] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.326426 2779 raft_consensus.cc:729] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.326443 2904 raft_consensus.cc:729] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.326545 2779 consensus_queue.cc:260] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.326553 2904 consensus_queue.cc:260] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.326819 2779 ts_tablet_manager.cc:1398] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:06.327739 2904 ts_tablet_manager.cc:1398] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:06.327904 2904 tablet_bootstrap.cc:492] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:06.328867 2904 tablet_bootstrap.cc:654] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.330296 2904 tablet_bootstrap.cc:492] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:06.330377 2904 ts_tablet_manager.cc:1367] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:06.330806 2904 raft_consensus.cc:348] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.330902 2904 raft_consensus.cc:374] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.330932 2904 raft_consensus.cc:729] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.331037 2904 consensus_queue.cc:260] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.331302 2904 ts_tablet_manager.cc:1398] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:06.347502 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:06.355283 2899 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:06.355607 2899 tablet.cc:1620] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:06.356719 2900 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:06.356768 2903 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:06.357024 2900 tablet.cc:1620] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:06.357035 2903 tablet.cc:1620] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: MemRowSet was empty: no flush needed.
I20240502 01:59:06.362341 24479 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=3ffbe59e53874c5da24a0b344a74fa8c] alter complete (version 2)
I20240502 01:59:06.410686 2785 raft_consensus.cc:482] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:06.410847 2785 raft_consensus.cc:504] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.411309 2785 leader_election.cc:288] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:06.411640 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae4216a686d494a988a6bce810f5eea" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:06.411684 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae4216a686d494a988a6bce810f5eea" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:06.411788 24897 raft_consensus.cc:2454] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:06.411823 27177 raft_consensus.cc:2454] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:06.412118 24701 leader_election.cc:302] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:06.412285 2785 raft_consensus.cc:2781] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:06.412344 2785 raft_consensus.cc:482] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:06.412370 2785 raft_consensus.cc:3037] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.413424 2785 raft_consensus.cc:504] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.413762 2785 leader_election.cc:288] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:06.414098 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae4216a686d494a988a6bce810f5eea" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:06.414139 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae4216a686d494a988a6bce810f5eea" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:06.414229 27177 raft_consensus.cc:3037] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.414248 24897 raft_consensus.cc:3037] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.415071 24897 raft_consensus.cc:2454] T eae4216a686d494a988a6bce810f5eea P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:06.415096 27177 raft_consensus.cc:2454] T eae4216a686d494a988a6bce810f5eea P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:06.415331 24701 leader_election.cc:302] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:06.415611 2785 raft_consensus.cc:2781] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:06.415692 2785 raft_consensus.cc:686] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:59:06.416000 2785 consensus_queue.cc:237] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:06.427131 24486 catalog_manager.cc:5484] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:06.457181 2767 raft_consensus.cc:482] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:06.457315 2767 raft_consensus.cc:504] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.457711 2767 leader_election.cc:288] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:06.458055 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea3d358fc7a4baea6dc66ce8373d0aa" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:06.458061 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea3d358fc7a4baea6dc66ce8373d0aa" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:06.458181 27446 raft_consensus.cc:2454] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:06.458182 25032 raft_consensus.cc:2454] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:06.458609 24564 leader_election.cc:302] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:06.458932 2767 raft_consensus.cc:2781] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:06.459003 2767 raft_consensus.cc:482] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:06.459046 2767 raft_consensus.cc:3037] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.460220 2767 raft_consensus.cc:504] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.460578 2767 leader_election.cc:288] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:06.460870 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea3d358fc7a4baea6dc66ce8373d0aa" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:06.460978 25032 raft_consensus.cc:3037] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.461017 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea3d358fc7a4baea6dc66ce8373d0aa" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:06.461134 27446 raft_consensus.cc:3037] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.462067 25032 raft_consensus.cc:2454] T fea3d358fc7a4baea6dc66ce8373d0aa P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:06.462191 27446 raft_consensus.cc:2454] T fea3d358fc7a4baea6dc66ce8373d0aa P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:06.462371 24564 leader_election.cc:302] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:06.465293 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:06.466538 2630 raft_consensus.cc:2781] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:06.466642 2630 raft_consensus.cc:686] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:06.466768 2630 consensus_queue.cc:237] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.469694 24482 catalog_manager.cc:5484] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:06.470937 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:06.473142 2899 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:06.473632 2903 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:06.477141 2900 tablet.cc:1722] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:06.480580 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=3ffbe59e53874c5da24a0b344a74fa8c] alter complete (version 3)
W20240502 01:59:06.485327 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:06.512979 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:06.521762 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:06.579212 2813 raft_consensus.cc:482] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:06.579336 2813 raft_consensus.cc:504] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.579718 2813 leader_election.cc:288] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:06.580154 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18207e699e08468c96fe5d99bc20e3c9" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:06.580296 24632 raft_consensus.cc:2454] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:06.580438 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18207e699e08468c96fe5d99bc20e3c9" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:06.580561 24897 raft_consensus.cc:2454] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 0.
I20240502 01:59:06.580843 27112 leader_election.cc:302] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:06.581144 2813 raft_consensus.cc:2781] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:06.581197 2813 raft_consensus.cc:482] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:06.581229 2813 raft_consensus.cc:3037] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.582659 2813 raft_consensus.cc:504] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.582998 2813 leader_election.cc:288] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:06.583355 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18207e699e08468c96fe5d99bc20e3c9" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:06.583505 24897 raft_consensus.cc:3037] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.583469 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18207e699e08468c96fe5d99bc20e3c9" candidate_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:06.583582 24632 raft_consensus.cc:3037] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:06.583799 24486 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:06.583907 24486 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:06.584538 24897 raft_consensus.cc:2454] T 18207e699e08468c96fe5d99bc20e3c9 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:06.584540 24632 raft_consensus.cc:2454] T 18207e699e08468c96fe5d99bc20e3c9 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd619a595e94004ac0f7f5a9054e7cc in term 1.
I20240502 01:59:06.585011 27112 leader_election.cc:302] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:06.585350 2812 raft_consensus.cc:2781] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:06.585422 2812 raft_consensus.cc:686] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Running, Role: LEADER
I20240502 01:59:06.585551 2812 consensus_queue.cc:237] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:06.587463 24486 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 4990271fc1144f5da2539a78057e2143
I20240502 01:59:06.588042 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4990271fc1144f5da2539a78057e2143 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:06 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:59:06.588251 24481 catalog_manager.cc:5484] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5). New cstate: current_term: 1 leader_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:06.588454 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet 4990271fc1144f5da2539a78057e2143 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:06 UTC) from {username='slave'} at 127.0.0.1:46448
I20240502 01:59:06.588811 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4990271fc1144f5da2539a78057e2143 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:06 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:06.589386 2966 tablet_replica.cc:330] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:06.589530 2966 raft_consensus.cc:2229] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:06.589645 2966 raft_consensus.cc:2258] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:06.590073 2963 tablet_replica.cc:330] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:59:06.590266 2963 raft_consensus.cc:2229] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:06.590523 2963 raft_consensus.cc:2258] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:06.590626 2966 ts_tablet_manager.cc:1875] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:06.591084 2963 ts_tablet_manager.cc:1875] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:06.593263 2963 ts_tablet_manager.cc:1888] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:06.593303 2966 ts_tablet_manager.cc:1888] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:06.593379 2966 log.cc:1196] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/4990271fc1144f5da2539a78057e2143
I20240502 01:59:06.593405 2963 log.cc:1196] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/4990271fc1144f5da2539a78057e2143
I20240502 01:59:06.593899 2963 ts_tablet_manager.cc:1909] T 4990271fc1144f5da2539a78057e2143 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:59:06.594849 2966 ts_tablet_manager.cc:1909] T 4990271fc1144f5da2539a78057e2143 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:06.594945 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table_new_name [id=3ffbe59e53874c5da24a0b344a74fa8c]) successfully deleted
I20240502 01:59:06.597370 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table_new_name [id=3ffbe59e53874c5da24a0b344a74fa8c]) successfully deleted
I20240502 01:59:06.600212 2964 tablet_replica.cc:330] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: stopping tablet replica
I20240502 01:59:06.606110 2964 raft_consensus.cc:2229] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:06.606314 2964 raft_consensus.cc:2258] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:06.607048 2964 ts_tablet_manager.cc:1875] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:06.609558 2964 ts_tablet_manager.cc:1888] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:06.609635 2964 log.cc:1196] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-4/wal/wals/4990271fc1144f5da2539a78057e2143
I20240502 01:59:06.610076 2964 ts_tablet_manager.cc:1909] T 4990271fc1144f5da2539a78057e2143 P 9dd619a595e94004ac0f7f5a9054e7cc: Deleting consensus metadata
I20240502 01:59:06.610802 24473 catalog_manager.cc:4830] TS 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239): tablet 4990271fc1144f5da2539a78057e2143 (table rebalancer_dynamic_table_new_name [id=3ffbe59e53874c5da24a0b344a74fa8c]) successfully deleted
W20240502 01:59:06.665119 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 01:59:06.665983 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet 252502ad2e124996b345888706e674d5: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:06.675660 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:06.690335 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:06.695505 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbd66f123ed448daf01bae177650752 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]), partition=
I20240502 01:59:06.695685 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbd66f123ed448daf01bae177650752 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]), partition=
I20240502 01:59:06.695788 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbd66f123ed448daf01bae177650752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.695950 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbd66f123ed448daf01bae177650752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.697721 2779 tablet_bootstrap.cc:492] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:06.697870 2787 tablet_bootstrap.cc:492] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:06.698117 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbd66f123ed448daf01bae177650752 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]), partition=
I20240502 01:59:06.698442 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbd66f123ed448daf01bae177650752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:06.698740 2787 tablet_bootstrap.cc:654] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.698920 2779 tablet_bootstrap.cc:654] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.699751 2787 tablet_bootstrap.cc:492] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:06.699836 2787 ts_tablet_manager.cc:1367] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:06.700241 2787 raft_consensus.cc:348] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.700343 2787 raft_consensus.cc:374] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.700376 2787 raft_consensus.cc:729] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.700496 2787 consensus_queue.cc:260] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.700587 2779 tablet_bootstrap.cc:492] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:06.700671 2779 ts_tablet_manager.cc:1367] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:06.700757 2787 ts_tablet_manager.cc:1398] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:06.700920 2910 tablet_bootstrap.cc:492] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:06.701063 2779 raft_consensus.cc:348] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.701160 2779 raft_consensus.cc:374] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.701184 2779 raft_consensus.cc:729] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.701290 2779 consensus_queue.cc:260] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.701551 2779 ts_tablet_manager.cc:1398] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:06.701642 2910 tablet_bootstrap.cc:654] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:06.703405 2910 tablet_bootstrap.cc:492] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:06.703485 2910 ts_tablet_manager.cc:1367] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:06.703831 2910 raft_consensus.cc:348] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.703915 2910 raft_consensus.cc:374] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:06.703935 2910 raft_consensus.cc:729] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:06.704037 2910 consensus_queue.cc:260] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:06.704316 2910 ts_tablet_manager.cc:1398] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:06.797459 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:06.856632 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:06.892201 1736 consensus_queue.cc:1035] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.897284 3012 consensus_queue.cc:1035] T eae4216a686d494a988a6bce810f5eea P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:06.902292 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:06.971576 2813 consensus_queue.cc:1035] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:06.976111 2813 consensus_queue.cc:1035] T 18207e699e08468c96fe5d99bc20e3c9 P 9dd619a595e94004ac0f7f5a9054e7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.005335 3009 raft_consensus.cc:482] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.005479 3009 raft_consensus.cc:504] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:07.005904 3009 leader_election.cc:288] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:07.006203 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd66f123ed448daf01bae177650752" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:07.006253 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd66f123ed448daf01bae177650752" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:07.006332 24897 raft_consensus.cc:2454] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:07.006421 25031 raft_consensus.cc:2454] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:07.006592 27380 leader_election.cc:302] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:07.007901 3009 raft_consensus.cc:2781] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.007959 3009 raft_consensus.cc:482] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.007987 3009 raft_consensus.cc:3037] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.008989 3009 raft_consensus.cc:504] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:07.009351 3009 leader_election.cc:288] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:07.010217 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd66f123ed448daf01bae177650752" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:07.010294 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd66f123ed448daf01bae177650752" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:07.010339 25031 raft_consensus.cc:3037] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.010408 24897 raft_consensus.cc:3037] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.011114 25031 raft_consensus.cc:2454] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:07.011144 24897 raft_consensus.cc:2454] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:07.011386 27380 leader_election.cc:302] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:07.011565 3009 raft_consensus.cc:2781] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.011649 3009 raft_consensus.cc:686] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:07.011781 3009 consensus_queue.cc:237] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } }
I20240502 01:59:07.014185 24486 catalog_manager.cc:5484] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:07.016525 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.031913 2946 consensus_queue.cc:1035] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.035821 2946 consensus_queue.cc:1035] T fea3d358fc7a4baea6dc66ce8373d0aa P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:07.044752 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:07.090996 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:07.110409 24481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:07.116720 24610 tablet_service.cc:1467] Processing CreateTablet for tablet acab5c782c794e70bd1b8c69c4749455 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.116720 27557 tablet_service.cc:1467] Processing CreateTablet for tablet acab5c782c794e70bd1b8c69c4749455 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.116777 24611 tablet_service.cc:1467] Processing CreateTablet for tablet c18a75a65f7e49c8b7cab6f54636dcf0 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.117024 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acab5c782c794e70bd1b8c69c4749455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.117024 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acab5c782c794e70bd1b8c69c4749455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.117180 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18a75a65f7e49c8b7cab6f54636dcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.117234 24744 tablet_service.cc:1467] Processing CreateTablet for tablet acab5c782c794e70bd1b8c69c4749455 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.117491 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acab5c782c794e70bd1b8c69c4749455. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.120127 24877 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd741190834b0b880f852523e52f49 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.120296 24744 tablet_service.cc:1467] Processing CreateTablet for tablet c18a75a65f7e49c8b7cab6f54636dcf0 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.120411 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd741190834b0b880f852523e52f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.120362 27423 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd741190834b0b880f852523e52f49 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.120517 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18a75a65f7e49c8b7cab6f54636dcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.120595 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd741190834b0b880f852523e52f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.120711 3049 tablet_bootstrap.cc:492] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:07.121810 3049 tablet_bootstrap.cc:654] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.122658 2779 tablet_bootstrap.cc:492] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:07.122887 27156 tablet_service.cc:1467] Processing CreateTablet for tablet c18a75a65f7e49c8b7cab6f54636dcf0 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.123062 3048 tablet_bootstrap.cc:492] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:07.123153 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18a75a65f7e49c8b7cab6f54636dcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.123418 2910 tablet_bootstrap.cc:492] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:07.123525 2779 tablet_bootstrap.cc:654] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.123950 3048 tablet_bootstrap.cc:654] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.124337 2910 tablet_bootstrap.cc:654] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.124801 2779 tablet_bootstrap.cc:492] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:07.124884 2779 ts_tablet_manager.cc:1367] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:07.125200 3048 tablet_bootstrap.cc:492] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:07.125277 3048 ts_tablet_manager.cc:1367] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:07.125361 2779 raft_consensus.cc:348] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.125465 2779 raft_consensus.cc:374] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.125495 2779 raft_consensus.cc:729] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.125603 2779 consensus_queue.cc:260] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.125707 3048 raft_consensus.cc:348] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.125788 3048 raft_consensus.cc:374] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.125808 3048 raft_consensus.cc:729] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.125888 2779 ts_tablet_manager.cc:1398] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.125902 3048 consensus_queue.cc:260] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.126184 3048 ts_tablet_manager.cc:1398] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.126360 3048 tablet_bootstrap.cc:492] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:07.126900 24610 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd741190834b0b880f852523e52f49 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=929caef6a4db4bf39aee7b9a361dac1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.127125 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd741190834b0b880f852523e52f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.127300 3048 tablet_bootstrap.cc:654] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.128839 3049 tablet_bootstrap.cc:492] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:07.128983 3049 ts_tablet_manager.cc:1367] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:59:07.129540 3049 raft_consensus.cc:348] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.129720 3049 raft_consensus.cc:374] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.129802 3049 raft_consensus.cc:729] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.129990 3055 tablet_bootstrap.cc:492] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:07.130163 3049 consensus_queue.cc:260] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
W20240502 01:59:07.130453 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.130470 3049 ts_tablet_manager.cc:1398] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.131225 3048 tablet_bootstrap.cc:492] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:07.131325 3048 ts_tablet_manager.cc:1367] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:07.131439 3055 tablet_bootstrap.cc:654] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.131845 3050 tablet_bootstrap.cc:492] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:07.132511 2910 tablet_bootstrap.cc:492] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:07.132603 2910 ts_tablet_manager.cc:1367] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:59:07.131907 3048 raft_consensus.cc:348] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.132685 3048 raft_consensus.cc:374] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.132725 3048 raft_consensus.cc:729] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.132864 3048 consensus_queue.cc:260] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.133090 3050 tablet_bootstrap.cc:654] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.133081 2910 raft_consensus.cc:348] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.133181 2910 raft_consensus.cc:374] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.133236 2910 raft_consensus.cc:729] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.133268 3048 ts_tablet_manager.cc:1398] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:07.133347 2910 consensus_queue.cc:260] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.133428 3048 tablet_bootstrap.cc:492] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:07.133608 2910 ts_tablet_manager.cc:1398] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.134420 3048 tablet_bootstrap.cc:654] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.134691 3055 tablet_bootstrap.cc:492] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:07.134779 3055 ts_tablet_manager.cc:1367] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:07.135200 3055 raft_consensus.cc:348] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.135293 3055 raft_consensus.cc:374] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.135320 3055 raft_consensus.cc:729] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.135413 3055 consensus_queue.cc:260] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.135696 3055 ts_tablet_manager.cc:1398] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.140076 3048 tablet_bootstrap.cc:492] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:07.140201 3048 ts_tablet_manager.cc:1367] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:59:07.140661 3048 raft_consensus.cc:348] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.140791 3048 raft_consensus.cc:374] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.140844 3048 raft_consensus.cc:729] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.140974 3048 consensus_queue.cc:260] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.141258 3048 ts_tablet_manager.cc:1398] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.141709 3050 tablet_bootstrap.cc:492] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:07.141793 3050 ts_tablet_manager.cc:1367] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:59:07.142293 3050 raft_consensus.cc:348] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.142812 3050 raft_consensus.cc:374] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.142853 3050 raft_consensus.cc:729] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.142989 3050 consensus_queue.cc:260] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.143297 3050 ts_tablet_manager.cc:1398] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.143529 3050 tablet_bootstrap.cc:492] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:07.144325 3050 tablet_bootstrap.cc:654] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.145972 3050 tablet_bootstrap.cc:492] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:07.146063 3050 ts_tablet_manager.cc:1367] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:07.146512 3050 raft_consensus.cc:348] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.146605 3050 raft_consensus.cc:374] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.147226 3050 raft_consensus.cc:729] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.147400 3050 consensus_queue.cc:260] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.147807 3050 ts_tablet_manager.cc:1398] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:07.160624 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:07.165275 25032 raft_consensus.cc:1261] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:07.165308 24897 raft_consensus.cc:1261] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Refusing update from remote peer 1cc0a6338e0a467dbdc6efb3e67a6a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:07.165658 3009 consensus_queue.cc:1035] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.165858 2730 consensus_queue.cc:1035] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.169364 24486 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]). Expected version 1 got 0
I20240502 01:59:07.170282 24481 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]). Expected version 1 got 0
I20240502 01:59:07.170329 3064 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:07.170646 3064 tablet.cc:1620] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:07.171113 24486 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]). Expected version 1 got 0
I20240502 01:59:07.171751 24481 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]). Expected version 1 got 0
I20240502 01:59:07.171862 3063 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:07.172150 3063 tablet.cc:1620] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:07.173133 3065 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:07.173518 3065 tablet.cc:1620] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:07.175417 24482 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c]). Expected version 1 got 0
I20240502 01:59:07.176533 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c] alter complete (version 1)
I20240502 01:59:07.178709 3012 raft_consensus.cc:482] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.178817 3012 raft_consensus.cc:504] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.179234 3012 leader_election.cc:288] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:59:07.179533 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acab5c782c794e70bd1b8c69c4749455" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:07.179672 24632 raft_consensus.cc:2454] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:07.179790 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acab5c782c794e70bd1b8c69c4749455" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:07.179919 27577 raft_consensus.cc:2454] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:07.179919 24700 leader_election.cc:302] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:07.180379 3012 raft_consensus.cc:2781] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.180434 3012 raft_consensus.cc:482] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.180469 3012 raft_consensus.cc:3037] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.181304 3012 raft_consensus.cc:504] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.181614 3012 leader_election.cc:288] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673)
I20240502 01:59:07.181988 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acab5c782c794e70bd1b8c69c4749455" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:07.182039 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acab5c782c794e70bd1b8c69c4749455" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:07.182101 24632 raft_consensus.cc:3037] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.182159 27577 raft_consensus.cc:3037] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.183136 27577 raft_consensus.cc:2454] T acab5c782c794e70bd1b8c69c4749455 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:07.183148 24632 raft_consensus.cc:2454] T acab5c782c794e70bd1b8c69c4749455 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:07.183562 24701 leader_election.cc:302] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:07.183883 3012 raft_consensus.cc:2781] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.183951 3012 raft_consensus.cc:686] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:59:07.184069 3012 consensus_queue.cc:237] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } }
I20240502 01:59:07.186321 24482 catalog_manager.cc:5484] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:07.239252 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:07.250504 2946 raft_consensus.cc:482] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.250633 2946 raft_consensus.cc:504] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.251045 2946 leader_election.cc:288] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:07.251406 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd741190834b0b880f852523e52f49" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:07.251550 27444 raft_consensus.cc:2454] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:07.251710 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd741190834b0b880f852523e52f49" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:07.251848 24897 raft_consensus.cc:2454] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:07.251859 24567 leader_election.cc:302] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:59:07.252138 2946 raft_consensus.cc:2781] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.252198 2946 raft_consensus.cc:482] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.252254 2946 raft_consensus.cc:3037] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.253219 2946 raft_consensus.cc:504] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.253496 2946 leader_election.cc:288] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:07.253813 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd741190834b0b880f852523e52f49" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:07.253913 27444 raft_consensus.cc:3037] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.254014 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd741190834b0b880f852523e52f49" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:07.254124 24897 raft_consensus.cc:3037] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.254747 27444 raft_consensus.cc:2454] T e6bd741190834b0b880f852523e52f49 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:07.255062 24897 raft_consensus.cc:2454] T e6bd741190834b0b880f852523e52f49 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:07.255373 24567 leader_election.cc:302] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:07.255734 2946 raft_consensus.cc:2781] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.255846 2946 raft_consensus.cc:686] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:07.256345 2946 consensus_queue.cc:237] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.258673 24482 catalog_manager.cc:5484] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:07.281277 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:07.287933 3063 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:07.288245 3063 tablet.cc:1620] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:07.289594 3064 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:07.289853 3064 tablet.cc:1620] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:07.291914 3065 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:07.292161 3065 tablet.cc:1620] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:07.297130 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=e62574175791410b8de084c1c14c923c] alter complete (version 2)
W20240502 01:59:07.297565 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:07.298142 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:07.331743 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:07.376029 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.381269 1736 raft_consensus.cc:482] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.381402 1736 raft_consensus.cc:504] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.381835 1736 leader_election.cc:288] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
W20240502 01:59:07.382346 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.382347 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18a75a65f7e49c8b7cab6f54636dcf0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:07.382416 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18a75a65f7e49c8b7cab6f54636dcf0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:07.382488 24632 raft_consensus.cc:2454] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:07.382560 27175 raft_consensus.cc:2454] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 0.
I20240502 01:59:07.382848 24700 leader_election.cc:302] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, 9dd619a595e94004ac0f7f5a9054e7cc; no voters:
I20240502 01:59:07.383018 1736 raft_consensus.cc:2781] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.383085 1736 raft_consensus.cc:482] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.383131 1736 raft_consensus.cc:3037] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.383962 1736 raft_consensus.cc:504] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.384258 1736 leader_election.cc:288] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Requested vote from peers 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:07.384596 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18a75a65f7e49c8b7cab6f54636dcf0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:07.384601 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18a75a65f7e49c8b7cab6f54636dcf0" candidate_uuid: "92cb785c165749eca49af5784756461c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:07.384696 24632 raft_consensus.cc:3037] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.384719 27175 raft_consensus.cc:3037] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.385643 24632 raft_consensus.cc:2454] T c18a75a65f7e49c8b7cab6f54636dcf0 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:07.385808 27175 raft_consensus.cc:2454] T c18a75a65f7e49c8b7cab6f54636dcf0 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cb785c165749eca49af5784756461c in term 1.
I20240502 01:59:07.385905 24700 leader_election.cc:302] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:07.386210 1736 raft_consensus.cc:2781] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.386291 1736 raft_consensus.cc:686] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [term 1 LEADER]: Becoming Leader. State: Replica: 92cb785c165749eca49af5784756461c, State: Running, Role: LEADER
I20240502 01:59:07.386420 1736 consensus_queue.cc:237] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.388733 24481 catalog_manager.cc:5484] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cb785c165749eca49af5784756461c (127.17.243.2). New cstate: current_term: 1 leader_uuid: "92cb785c165749eca49af5784756461c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:07.399791 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:07.404847 3063 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:07.407275 3065 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:07.408336 3064 tablet.cc:1722] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:07.412781 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=e62574175791410b8de084c1c14c923c] alter complete (version 3)
W20240502 01:59:07.425257 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.515164 24482 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:07.515313 24482 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:07.517406 24482 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 7cbd66f123ed448daf01bae177650752
I20240502 01:59:07.517975 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbd66f123ed448daf01bae177650752 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:07 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:59:07.518020 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbd66f123ed448daf01bae177650752 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:07 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:59:07.518283 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbd66f123ed448daf01bae177650752 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:07 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:07.518347 3100 tablet_replica.cc:330] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:59:07.518342 3101 tablet_replica.cc:330] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:59:07.518487 3101 raft_consensus.cc:2229] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:07.518496 3100 raft_consensus.cc:2229] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:07.518623 3100 raft_consensus.cc:2258] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:07.518699 3102 tablet_replica.cc:330] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:07.518836 3102 raft_consensus.cc:2229] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:07.518947 3102 raft_consensus.cc:2258] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:07.518697 3101 raft_consensus.cc:2258] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:07.519447 3101 ts_tablet_manager.cc:1875] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:07.519565 3100 ts_tablet_manager.cc:1875] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:07.520066 3102 ts_tablet_manager.cc:1875] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:07.521806 3100 ts_tablet_manager.cc:1888] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:07.521963 3100 log.cc:1196] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/7cbd66f123ed448daf01bae177650752
I20240502 01:59:07.521986 3102 ts_tablet_manager.cc:1888] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:07.522066 3102 log.cc:1196] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/7cbd66f123ed448daf01bae177650752
I20240502 01:59:07.522344 3100 ts_tablet_manager.cc:1909] T 7cbd66f123ed448daf01bae177650752 P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:59:07.522970 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table_new_name [id=e62574175791410b8de084c1c14c923c]) successfully deleted
I20240502 01:59:07.523901 3101 ts_tablet_manager.cc:1888] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:07.523968 3101 log.cc:1196] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/7cbd66f123ed448daf01bae177650752
I20240502 01:59:07.524211 3101 ts_tablet_manager.cc:1909] T 7cbd66f123ed448daf01bae177650752 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:59:07.524874 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table_new_name [id=e62574175791410b8de084c1c14c923c]) successfully deleted
I20240502 01:59:07.525069 3102 ts_tablet_manager.cc:1909] T 7cbd66f123ed448daf01bae177650752 P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:07.525678 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 7cbd66f123ed448daf01bae177650752 (table rebalancer_dynamic_table_new_name [id=e62574175791410b8de084c1c14c923c]) successfully deleted
W20240502 01:59:07.610503 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.619387 24482 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:07.625090 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 6acb791d135b49eb9a6c8018ebec602d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c]), partition=
I20240502 01:59:07.625386 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acb791d135b49eb9a6c8018ebec602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.627296 2910 tablet_bootstrap.cc:492] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:07.627502 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 6acb791d135b49eb9a6c8018ebec602d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c]), partition=
I20240502 01:59:07.627724 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acb791d135b49eb9a6c8018ebec602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.628218 2910 tablet_bootstrap.cc:654] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.629294 2910 tablet_bootstrap.cc:492] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:07.629354 2910 ts_tablet_manager.cc:1367] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:07.629626 3130 tablet_bootstrap.cc:492] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:07.629778 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 6acb791d135b49eb9a6c8018ebec602d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c]), partition=
I20240502 01:59:07.629750 2910 raft_consensus.cc:348] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.629853 2910 raft_consensus.cc:374] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.629885 2910 raft_consensus.cc:729] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.630025 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acb791d135b49eb9a6c8018ebec602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.630018 2910 consensus_queue.cc:260] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.630292 2910 ts_tablet_manager.cc:1398] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.630630 3130 tablet_bootstrap.cc:654] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.632369 3132 tablet_bootstrap.cc:492] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:07.633200 3130 tablet_bootstrap.cc:492] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:07.633297 3130 ts_tablet_manager.cc:1367] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:59:07.635051 3132 tablet_bootstrap.cc:654] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.633749 3130 raft_consensus.cc:348] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.635373 3130 raft_consensus.cc:374] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.635426 3130 raft_consensus.cc:729] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.635557 3130 consensus_queue.cc:260] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.635818 3130 ts_tablet_manager.cc:1398] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:07.636389 3132 tablet_bootstrap.cc:492] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:07.636485 3132 ts_tablet_manager.cc:1367] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:07.637033 3132 raft_consensus.cc:348] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.637123 3132 raft_consensus.cc:374] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.637143 3132 raft_consensus.cc:729] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.637261 3132 consensus_queue.cc:260] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.637516 3132 ts_tablet_manager.cc:1398] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:07.655403 1736 consensus_queue.cc:1035] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.658614 2785 consensus_queue.cc:1035] T acab5c782c794e70bd1b8c69c4749455 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.676900 3067 consensus_queue.cc:1035] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.682848 3140 consensus_queue.cc:1035] T e6bd741190834b0b880f852523e52f49 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:07.683681 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.700616 2261 raft_consensus.cc:482] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.700719 2261 raft_consensus.cc:504] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.701064 2261 leader_election.cc:288] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:07.701361 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acb791d135b49eb9a6c8018ebec602d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:07.701465 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acb791d135b49eb9a6c8018ebec602d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:07.701517 27577 raft_consensus.cc:2454] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:07.701594 27446 raft_consensus.cc:2454] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 0.
I20240502 01:59:07.701893 24833 leader_election.cc:302] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:07.702100 2261 raft_consensus.cc:2781] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.702162 2261 raft_consensus.cc:482] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.702204 2261 raft_consensus.cc:3037] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.703361 2261 raft_consensus.cc:504] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.703624 2261 leader_election.cc:288] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021)
I20240502 01:59:07.703832 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acb791d135b49eb9a6c8018ebec602d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:07.703943 27577 raft_consensus.cc:3037] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.703994 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acb791d135b49eb9a6c8018ebec602d" candidate_uuid: "f46bcd2bc124471981349b1e2726253b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:07.704102 27446 raft_consensus.cc:3037] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.704922 27577 raft_consensus.cc:2454] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:07.705005 27446 raft_consensus.cc:2454] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46bcd2bc124471981349b1e2726253b in term 1.
I20240502 01:59:07.705214 24833 leader_election.cc:302] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:07.705372 2261 raft_consensus.cc:2781] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.705467 2261 raft_consensus.cc:686] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Becoming Leader. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Running, Role: LEADER
I20240502 01:59:07.705597 2261 consensus_queue.cc:237] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:07.707676 24486 catalog_manager.cc:5484] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b reported cstate change: term changed from 0 to 1, leader changed from <none> to f46bcd2bc124471981349b1e2726253b (127.17.243.3). New cstate: current_term: 1 leader_uuid: "f46bcd2bc124471981349b1e2726253b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: HEALTHY } } }
W20240502 01:59:07.720088 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:07.781522 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.823192 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:07.827730 27444 raft_consensus.cc:1261] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:07.827795 27578 raft_consensus.cc:1261] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer f46bcd2bc124471981349b1e2726253b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:07.828207 3143 consensus_queue.cc:1035] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.828372 2261 consensus_queue.cc:1035] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:07.830528 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.831187 3064 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:07.831450 3064 tablet.cc:1620] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:07.832429 3063 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:07.832710 3063 tablet.cc:1620] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:07.833568 24486 catalog_manager.cc:5576] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445) does not have the latest schema for tablet 6acb791d135b49eb9a6c8018ebec602d (table rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c]). Expected version 1 got 0
I20240502 01:59:07.834399 24481 catalog_manager.cc:5576] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021) does not have the latest schema for tablet 6acb791d135b49eb9a6c8018ebec602d (table rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c]). Expected version 1 got 0
I20240502 01:59:07.835615 24482 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 6acb791d135b49eb9a6c8018ebec602d (table rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c]). Expected version 1 got 0
I20240502 01:59:07.836004 3172 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:07.836516 3172 tablet.cc:1620] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:07.841711 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c] alter complete (version 1)
W20240502 01:59:07.894613 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:07.921056 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:07.923990 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.929560 24486 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:07.936225 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 0922c0a1f5d6475cac76629f88ec337d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.936522 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0922c0a1f5d6475cac76629f88ec337d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.936602 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 0922c0a1f5d6475cac76629f88ec337d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.936893 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0922c0a1f5d6475cac76629f88ec337d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.938513 3173 tablet_bootstrap.cc:492] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:07.939065 3174 tablet_bootstrap.cc:492] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:07.939529 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 0922c0a1f5d6475cac76629f88ec337d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.939787 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0922c0a1f5d6475cac76629f88ec337d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.939999 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 98d5d22f9b254447b920ae04284e6fc4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.939529 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 98d5d22f9b254447b920ae04284e6fc4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.940269 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d5d22f9b254447b920ae04284e6fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.940289 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d5d22f9b254447b920ae04284e6fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.941138 3173 tablet_bootstrap.cc:654] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.942544 2910 tablet_bootstrap.cc:492] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:07.943342 3174 tablet_bootstrap.cc:654] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.943557 2910 tablet_bootstrap.cc:654] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.944095 3173 tablet_bootstrap.cc:492] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:07.944190 3173 ts_tablet_manager.cc:1367] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:07.944599 3174 tablet_bootstrap.cc:492] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:07.944682 3174 ts_tablet_manager.cc:1367] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:07.945056 3174 raft_consensus.cc:348] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.945077 3173 raft_consensus.cc:348] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.945235 3173 raft_consensus.cc:374] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.945262 3174 raft_consensus.cc:374] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.945319 3173 raft_consensus.cc:729] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.945467 3174 raft_consensus.cc:729] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.945536 3173 consensus_queue.cc:260] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.945557 24486 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:07.945725 3174 consensus_queue.cc:260] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.945969 2910 tablet_bootstrap.cc:492] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:07.946049 2910 ts_tablet_manager.cc:1367] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:07.946137 3174 ts_tablet_manager.cc:1398] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.946455 2910 raft_consensus.cc:348] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:07.946552 2910 raft_consensus.cc:374] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.946578 2910 raft_consensus.cc:729] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.946677 2910 consensus_queue.cc:260] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:07.946928 2910 ts_tablet_manager.cc:1398] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.947013 3173 ts_tablet_manager.cc:1398] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240502 01:59:07.950467 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.950803 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 98d5d22f9b254447b920ae04284e6fc4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.951033 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d5d22f9b254447b920ae04284e6fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.953087 3174 tablet_bootstrap.cc:492] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:07.954138 3174 tablet_bootstrap.cc:654] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.954178 27423 tablet_service.cc:1467] Processing CreateTablet for tablet ab4356c5a69f47dabd2ededabc9b6ef1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.954460 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4356c5a69f47dabd2ededabc9b6ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.954141 25012 tablet_service.cc:1467] Processing CreateTablet for tablet ab4356c5a69f47dabd2ededabc9b6ef1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.955276 3174 tablet_bootstrap.cc:492] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:07.955444 3174 ts_tablet_manager.cc:1367] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:07.955502 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4356c5a69f47dabd2ededabc9b6ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.955888 3174 raft_consensus.cc:348] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:07.956043 3174 raft_consensus.cc:374] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.956092 3174 raft_consensus.cc:729] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.956230 3174 consensus_queue.cc:260] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:07.956564 3174 ts_tablet_manager.cc:1398] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:07.959297 3174 tablet_bootstrap.cc:492] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:07.960201 3174 tablet_bootstrap.cc:654] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.960584 3064 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:07.960886 3064 tablet.cc:1620] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: MemRowSet was empty: no flush needed.
I20240502 01:59:07.960950 3176 tablet_bootstrap.cc:492] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:07.961944 3176 tablet_bootstrap.cc:654] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.965047 3172 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:07.965129 3063 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:07.965351 3172 tablet.cc:1620] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:07.965358 3063 tablet.cc:1620] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: MemRowSet was empty: no flush needed.
I20240502 01:59:07.968500 24486 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=7f7fcc72fdba442fa268d179fdb8a60c] alter complete (version 2)
I20240502 01:59:07.969498 1736 consensus_queue.cc:1035] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.973094 27156 tablet_service.cc:1467] Processing CreateTablet for tablet ab4356c5a69f47dabd2ededabc9b6ef1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fef7c6f5c76a4462b4b9d8859a081122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:07.973377 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4356c5a69f47dabd2ededabc9b6ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.974511 1891 raft_consensus.cc:482] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.974597 1891 raft_consensus.cc:504] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.974889 3012 consensus_queue.cc:1035] T c18a75a65f7e49c8b7cab6f54636dcf0 P 92cb785c165749eca49af5784756461c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:07.975324 3173 tablet_bootstrap.cc:492] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:07.975298 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0922c0a1f5d6475cac76629f88ec337d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:07.976311 3173 tablet_bootstrap.cc:654] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
W20240502 01:59:07.976379 24966 leader_election.cc:341] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92cb785c165749eca49af5784756461c (127.17.243.2:45237): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:59:07.977607 1891 leader_election.cc:288] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:07.977988 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0922c0a1f5d6475cac76629f88ec337d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:07.978101 27177 raft_consensus.cc:2454] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:07.978616 2910 tablet_bootstrap.cc:492] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:07.980140 3173 tablet_bootstrap.cc:492] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:07.980227 3173 ts_tablet_manager.cc:1367] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:07.980724 3173 raft_consensus.cc:348] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.980854 3173 raft_consensus.cc:374] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.980865 3176 tablet_bootstrap.cc:492] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:07.980890 3173 raft_consensus.cc:729] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.980943 3176 ts_tablet_manager.cc:1367] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 01:59:07.981020 3173 consensus_queue.cc:260] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.981333 3173 ts_tablet_manager.cc:1398] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.981350 3176 raft_consensus.cc:348] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.981431 3176 raft_consensus.cc:374] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.981451 3176 raft_consensus.cc:729] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.981654 3176 consensus_queue.cc:260] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.982049 3176 ts_tablet_manager.cc:1398] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.982239 3176 tablet_bootstrap.cc:492] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:07.983258 3176 tablet_bootstrap.cc:654] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.983516 2910 tablet_bootstrap.cc:654] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:07.978466 24965 leader_election.cc:302] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters: 92cb785c165749eca49af5784756461c
I20240502 01:59:07.979753 3174 tablet_bootstrap.cc:492] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:07.985903 3180 raft_consensus.cc:2781] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.985985 3180 raft_consensus.cc:482] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.986007 3174 ts_tablet_manager.cc:1367] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.001s
I20240502 01:59:07.986022 3180 raft_consensus.cc:3037] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.986438 3174 raft_consensus.cc:348] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.986529 3174 raft_consensus.cc:374] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.986555 3174 raft_consensus.cc:729] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.986601 2910 tablet_bootstrap.cc:492] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:07.986682 2910 ts_tablet_manager.cc:1367] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:59:07.986665 3174 consensus_queue.cc:260] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.986835 3176 tablet_bootstrap.cc:492] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:07.986941 3174 ts_tablet_manager.cc:1398] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:07.987008 3180 raft_consensus.cc:504] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.987118 2910 raft_consensus.cc:348] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.987234 2910 raft_consensus.cc:374] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.987267 2910 raft_consensus.cc:729] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.987329 3180 leader_election.cc:288] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:07.987393 2910 consensus_queue.cc:260] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.987681 2910 ts_tablet_manager.cc:1398] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:07.987715 3176 ts_tablet_manager.cc:1367] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
W20240502 01:59:07.988973 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:07.989261 3180 raft_consensus.cc:482] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:07.989331 3180 raft_consensus.cc:504] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.989627 3180 leader_election.cc:288] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:07.989727 3176 raft_consensus.cc:348] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:07.989843 3176 raft_consensus.cc:374] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:07.989877 3176 raft_consensus.cc:729] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:07.989915 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0922c0a1f5d6475cac76629f88ec337d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:07.990053 27177 raft_consensus.cc:3037] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.990027 3176 consensus_queue.cc:260] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:07.990114 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4356c5a69f47dabd2ededabc9b6ef1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" is_pre_election: true
I20240502 01:59:07.990221 27444 raft_consensus.cc:2454] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:07.990298 3176 ts_tablet_manager.cc:1398] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:07.990491 24966 leader_election.cc:302] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:07.990669 3180 raft_consensus.cc:2781] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:07.990725 3180 raft_consensus.cc:482] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:07.990756 3180 raft_consensus.cc:3037] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.990969 27177 raft_consensus.cc:2454] T 0922c0a1f5d6475cac76629f88ec337d P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:07.991356 24965 leader_election.cc:302] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:07.991528 1891 raft_consensus.cc:2781] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.991603 1891 raft_consensus.cc:686] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:07.991670 3180 raft_consensus.cc:504] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.991722 1891 consensus_queue.cc:237] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:07.991988 3180 leader_election.cc:288] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:07.992451 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4356c5a69f47dabd2ededabc9b6ef1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09"
I20240502 01:59:07.992551 27444 raft_consensus.cc:3037] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.992564 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4356c5a69f47dabd2ededabc9b6ef1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:07.992671 27175 raft_consensus.cc:3037] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.993499 27444 raft_consensus.cc:2454] T ab4356c5a69f47dabd2ededabc9b6ef1 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:07.993594 27175 raft_consensus.cc:2454] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:07.993781 24966 leader_election.cc:302] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:07.993983 1891 raft_consensus.cc:2781] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:07.994062 1891 raft_consensus.cc:686] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:07.994179 1891 consensus_queue.cc:237] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:07.994771 24479 catalog_manager.cc:5484] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:07.997846 27177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4356c5a69f47dabd2ededabc9b6ef1" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:07.997988 27177 raft_consensus.cc:2362] T ab4356c5a69f47dabd2ededabc9b6ef1 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1. Re-sending same reply.
I20240502 01:59:07.998512 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0922c0a1f5d6475cac76629f88ec337d" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:07.998623 24765 raft_consensus.cc:3037] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:07.999715 24765 raft_consensus.cc:2454] T 0922c0a1f5d6475cac76629f88ec337d P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:08.002121 24479 catalog_manager.cc:5484] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:08.002842 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:08.013696 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:08.036134 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:08.042421 3099 raft_consensus.cc:482] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:08.042532 3099 raft_consensus.cc:504] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:08.042985 3099 leader_election.cc:288] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:08.043283 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d5d22f9b254447b920ae04284e6fc4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:08.043365 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d5d22f9b254447b920ae04284e6fc4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:08.043428 24765 raft_consensus.cc:2454] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:08.043488 25032 raft_consensus.cc:2454] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:08.043772 27377 leader_election.cc:302] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:08.044431 3099 raft_consensus.cc:2781] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:08.044498 3099 raft_consensus.cc:482] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:08.044540 3099 raft_consensus.cc:3037] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.045560 3099 raft_consensus.cc:504] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:08.045955 3099 leader_election.cc:288] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:08.046253 24765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d5d22f9b254447b920ae04284e6fc4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:08.046368 24765 raft_consensus.cc:3037] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.046569 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d5d22f9b254447b920ae04284e6fc4" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:08.046666 25032 raft_consensus.cc:3037] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.047225 24765 raft_consensus.cc:2454] T 98d5d22f9b254447b920ae04284e6fc4 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:08.047423 25032 raft_consensus.cc:2454] T 98d5d22f9b254447b920ae04284e6fc4 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:08.047693 27377 leader_election.cc:302] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:08.047868 3099 raft_consensus.cc:2781] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:08.047976 3099 raft_consensus.cc:686] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:08.048126 3099 consensus_queue.cc:237] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } }
I20240502 01:59:08.050704 24479 catalog_manager.cc:5484] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:08.071194 24479 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:08.076943 3064 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:08.077368 3172 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:08.077674 3063 tablet.cc:1722] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:08.083072 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=7f7fcc72fdba442fa268d179fdb8a60c] alter complete (version 3)
W20240502 01:59:08.102833 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:08.156864 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:08.184733 24487 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:08.184913 24487 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:08.187096 24487 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 6acb791d135b49eb9a6c8018ebec602d
I20240502 01:59:08.187677 27557 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acb791d135b49eb9a6c8018ebec602d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:08 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:59:08.187714 27423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acb791d135b49eb9a6c8018ebec602d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:08 UTC) from {username='slave'} at 127.0.0.1:36264
I20240502 01:59:08.187986 3220 tablet_replica.cc:330] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:59:08.188150 3220 raft_consensus.cc:2229] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:08.188145 24877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acb791d135b49eb9a6c8018ebec602d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:08 UTC) from {username='slave'} at 127.0.0.1:52444
I20240502 01:59:08.188272 3220 raft_consensus.cc:2258] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.188434 3222 tablet_replica.cc:330] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: stopping tablet replica
I20240502 01:59:08.188597 3222 raft_consensus.cc:2229] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:08.188812 3222 raft_consensus.cc:2258] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.188844 3220 ts_tablet_manager.cc:1875] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:08.189294 3222 ts_tablet_manager.cc:1875] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:08.191094 3222 ts_tablet_manager.cc:1888] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:08.191174 3222 log.cc:1196] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-2/wal/wals/6acb791d135b49eb9a6c8018ebec602d
I20240502 01:59:08.191234 3220 ts_tablet_manager.cc:1888] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:08.191332 3220 log.cc:1196] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/6acb791d135b49eb9a6c8018ebec602d
I20240502 01:59:08.191473 3222 ts_tablet_manager.cc:1909] T 6acb791d135b49eb9a6c8018ebec602d P f46bcd2bc124471981349b1e2726253b: Deleting consensus metadata
I20240502 01:59:08.191615 3220 ts_tablet_manager.cc:1909] T 6acb791d135b49eb9a6c8018ebec602d P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:59:08.192178 24474 catalog_manager.cc:4830] TS f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445): tablet 6acb791d135b49eb9a6c8018ebec602d (table rebalancer_dynamic_table_new_name [id=7f7fcc72fdba442fa268d179fdb8a60c]) successfully deleted
I20240502 01:59:08.192641 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 6acb791d135b49eb9a6c8018ebec602d (table rebalancer_dynamic_table_new_name [id=7f7fcc72fdba442fa268d179fdb8a60c]) successfully deleted
I20240502 01:59:08.194105 3221 tablet_replica.cc:330] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: stopping tablet replica
I20240502 01:59:08.194254 3221 raft_consensus.cc:2229] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:08.194365 3221 raft_consensus.cc:2258] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.194828 3221 ts_tablet_manager.cc:1875] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:08.196657 3221 ts_tablet_manager.cc:1888] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:08.196739 3221 log.cc:1196] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-6/wal/wals/6acb791d135b49eb9a6c8018ebec602d
I20240502 01:59:08.197041 3221 ts_tablet_manager.cc:1909] T 6acb791d135b49eb9a6c8018ebec602d P 1cc0a6338e0a467dbdc6efb3e67a6a09: Deleting consensus metadata
I20240502 01:59:08.197603 24474 catalog_manager.cc:4830] TS 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7:42021): tablet 6acb791d135b49eb9a6c8018ebec602d (table rebalancer_dynamic_table_new_name [id=7f7fcc72fdba442fa268d179fdb8a60c]) successfully deleted
W20240502 01:59:08.238345 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet da60bdbf6831421e9f9abec53c6e1d35: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:08.288717 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:08.294472 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 3d638889a39b4847960594bdf86079eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]), partition=
I20240502 01:59:08.294595 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 3d638889a39b4847960594bdf86079eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]), partition=
I20240502 01:59:08.294744 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d638889a39b4847960594bdf86079eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.294873 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d638889a39b4847960594bdf86079eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.296798 3174 tablet_bootstrap.cc:492] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:08.297042 3251 tablet_bootstrap.cc:492] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:08.297708 3174 tablet_bootstrap.cc:654] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.298115 3251 tablet_bootstrap.cc:654] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.298511 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 3d638889a39b4847960594bdf86079eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]), partition=
I20240502 01:59:08.298753 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d638889a39b4847960594bdf86079eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.300935 3253 tablet_bootstrap.cc:492] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:08.302868 3253 tablet_bootstrap.cc:654] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.303085 3251 tablet_bootstrap.cc:492] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:08.303210 3251 ts_tablet_manager.cc:1367] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:08.303686 3251 raft_consensus.cc:348] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.304011 3251 raft_consensus.cc:374] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.304103 3251 raft_consensus.cc:729] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.304507 3251 consensus_queue.cc:260] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.306612 3174 tablet_bootstrap.cc:492] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:08.306702 3174 ts_tablet_manager.cc:1367] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:59:08.306792 3253 tablet_bootstrap.cc:492] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:08.306890 3253 ts_tablet_manager.cc:1367] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:08.307183 3174 raft_consensus.cc:348] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.307281 3174 raft_consensus.cc:374] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.307307 3174 raft_consensus.cc:729] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.307416 3174 consensus_queue.cc:260] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.307682 3174 ts_tablet_manager.cc:1398] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:08.308486 3253 raft_consensus.cc:348] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.308691 3253 raft_consensus.cc:374] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.308926 3253 raft_consensus.cc:729] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.309067 3253 consensus_queue.cc:260] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.309442 3253 ts_tablet_manager.cc:1398] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:08.310837 3251 ts_tablet_manager.cc:1398] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
W20240502 01:59:08.354411 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:08.356240 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:08.439509 1891 consensus_queue.cc:1035] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.443676 3190 consensus_queue.cc:1035] T 0922c0a1f5d6475cac76629f88ec337d P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.491430 3067 raft_consensus.cc:482] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:08.491540 3067 raft_consensus.cc:504] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.491926 3067 leader_election.cc:288] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:08.492219 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d638889a39b4847960594bdf86079eb" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:08.492341 25032 raft_consensus.cc:2454] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:08.492739 24564 leader_election.cc:302] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:08.492991 3067 raft_consensus.cc:2781] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:08.493062 3067 raft_consensus.cc:482] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:08.493104 3067 raft_consensus.cc:3037] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.493427 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d638889a39b4847960594bdf86079eb" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:08.493549 27577 raft_consensus.cc:2454] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:08.494236 3067 raft_consensus.cc:504] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.494555 3067 leader_election.cc:288] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879)
I20240502 01:59:08.494792 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d638889a39b4847960594bdf86079eb" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:08.494904 27577 raft_consensus.cc:3037] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.494946 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d638889a39b4847960594bdf86079eb" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:08.495059 25032 raft_consensus.cc:3037] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.495842 27577 raft_consensus.cc:2454] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:08.496029 25032 raft_consensus.cc:2454] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:08.496271 24564 leader_election.cc:302] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:08.496439 3067 raft_consensus.cc:2781] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:08.496531 3067 raft_consensus.cc:686] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:08.496654 3067 consensus_queue.cc:237] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.499819 24479 catalog_manager.cc:5484] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:08.514333 3190 consensus_queue.cc:1035] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.519210 1891 consensus_queue.cc:1035] T ab4356c5a69f47dabd2ededabc9b6ef1 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:08.534118 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 01:59:08.549522 3099 consensus_queue.cc:1035] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.552307 3289 consensus_queue.cc:1035] T 98d5d22f9b254447b920ae04284e6fc4 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.568902 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:08.575457 27558 tablet_service.cc:1467] Processing CreateTablet for tablet 452b250aa44a48f197f681e5d1f34272 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.575739 27558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452b250aa44a48f197f681e5d1f34272. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.575762 27557 tablet_service.cc:1467] Processing CreateTablet for tablet 2eca9d656f5542d1855e727be59449e1 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.576005 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eca9d656f5542d1855e727be59449e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.575965 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 452b250aa44a48f197f681e5d1f34272 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.576206 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452b250aa44a48f197f681e5d1f34272. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.577692 3251 tablet_bootstrap.cc:492] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:08.577854 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8d9037cece45b49c4b0b399e62fd5f (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.578092 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f8d9037cece45b49c4b0b399e62fd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.578102 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 2eca9d656f5542d1855e727be59449e1 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.578305 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eca9d656f5542d1855e727be59449e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.578841 3251 tablet_bootstrap.cc:654] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.580150 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8d9037cece45b49c4b0b399e62fd5f (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.580435 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f8d9037cece45b49c4b0b399e62fd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.580621 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 2eca9d656f5542d1855e727be59449e1 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.580845 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eca9d656f5542d1855e727be59449e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.582278 3295 tablet_bootstrap.cc:492] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:08.583128 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8d9037cece45b49c4b0b399e62fd5f (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.583326 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 452b250aa44a48f197f681e5d1f34272 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=d197e389005343609110869935b96fce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:08.583403 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f8d9037cece45b49c4b0b399e62fd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.583597 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452b250aa44a48f197f681e5d1f34272. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:08.584317 3251 tablet_bootstrap.cc:492] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:08.584406 3251 ts_tablet_manager.cc:1367] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:59:08.584834 3251 raft_consensus.cc:348] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.584935 3251 raft_consensus.cc:374] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.584962 3251 raft_consensus.cc:729] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.585068 3251 consensus_queue.cc:260] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.585336 3251 ts_tablet_manager.cc:1398] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:08.585491 3251 tablet_bootstrap.cc:492] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:08.586287 3251 tablet_bootstrap.cc:654] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.586699 3295 tablet_bootstrap.cc:654] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
W20240502 01:59:08.586735 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:08.587777 3253 tablet_bootstrap.cc:492] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:08.588672 3253 tablet_bootstrap.cc:654] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.590626 3295 tablet_bootstrap.cc:492] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:08.590737 3295 ts_tablet_manager.cc:1367] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:08.591251 3295 raft_consensus.cc:348] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.591459 3295 raft_consensus.cc:374] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.591523 3295 raft_consensus.cc:729] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.591652 3295 consensus_queue.cc:260] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.591948 3295 ts_tablet_manager.cc:1398] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:08.592525 3295 tablet_bootstrap.cc:492] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:08.592939 3296 tablet_bootstrap.cc:492] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:08.593360 3295 tablet_bootstrap.cc:654] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.593871 3251 tablet_bootstrap.cc:492] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:08.593981 3251 ts_tablet_manager.cc:1367] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:08.594044 3293 tablet_bootstrap.cc:492] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:08.594400 3253 tablet_bootstrap.cc:492] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:08.594401 3251 raft_consensus.cc:348] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.594485 3253 ts_tablet_manager.cc:1367] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:59:08.594492 3251 raft_consensus.cc:374] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.594523 3251 raft_consensus.cc:729] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.594628 3251 consensus_queue.cc:260] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.594880 3251 ts_tablet_manager.cc:1398] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:08.594940 3253 raft_consensus.cc:348] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.595031 3253 raft_consensus.cc:374] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.595044 3251 tablet_bootstrap.cc:492] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:08.595060 3253 raft_consensus.cc:729] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.595063 3293 tablet_bootstrap.cc:654] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.595166 3253 consensus_queue.cc:260] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.595436 3253 ts_tablet_manager.cc:1398] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:08.595739 3296 tablet_bootstrap.cc:654] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.595845 3251 tablet_bootstrap.cc:654] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.598083 3296 tablet_bootstrap.cc:492] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:08.598173 3296 ts_tablet_manager.cc:1367] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:08.598868 3293 tablet_bootstrap.cc:492] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:08.598959 3293 ts_tablet_manager.cc:1367] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:08.599272 3296 raft_consensus.cc:348] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.599411 3296 raft_consensus.cc:374] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.599442 3296 raft_consensus.cc:729] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.601877 3251 tablet_bootstrap.cc:492] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:08.601976 3251 ts_tablet_manager.cc:1367] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:59:08.602319 3251 raft_consensus.cc:348] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.602397 3251 raft_consensus.cc:374] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.602425 3251 raft_consensus.cc:729] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.602794 3251 consensus_queue.cc:260] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.603112 3251 ts_tablet_manager.cc:1398] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:08.603935 3296 consensus_queue.cc:260] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.604207 3296 ts_tablet_manager.cc:1398] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:59:08.604718 3295 tablet_bootstrap.cc:492] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:08.604843 3295 ts_tablet_manager.cc:1367] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:59:08.604995 3293 raft_consensus.cc:348] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.605108 3293 raft_consensus.cc:374] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.605144 3293 raft_consensus.cc:729] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.605263 3293 consensus_queue.cc:260] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.605525 3295 raft_consensus.cc:348] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.605665 3295 raft_consensus.cc:374] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.605737 3295 raft_consensus.cc:729] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.605130 3289 raft_consensus.cc:482] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:08.606494 3293 ts_tablet_manager.cc:1398] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:59:08.606535 3289 raft_consensus.cc:504] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.606655 3293 tablet_bootstrap.cc:492] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:08.606858 3289 leader_election.cc:288] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:08.606881 3295 consensus_queue.cc:260] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.607311 3295 ts_tablet_manager.cc:1398] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:08.607385 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452b250aa44a48f197f681e5d1f34272" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:08.607486 24763 raft_consensus.cc:2454] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:08.607806 3293 tablet_bootstrap.cc:654] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:08.608371 27380 leader_election.cc:302] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:08.608736 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452b250aa44a48f197f681e5d1f34272" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:08.608834 27577 raft_consensus.cc:2454] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:08.609570 3289 raft_consensus.cc:2781] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:08.609623 3289 raft_consensus.cc:482] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:08.609658 3289 raft_consensus.cc:3037] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.610591 3289 raft_consensus.cc:504] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.610877 3289 leader_election.cc:288] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:08.611814 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452b250aa44a48f197f681e5d1f34272" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:08.611912 24763 raft_consensus.cc:3037] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.612865 24763 raft_consensus.cc:2454] T 452b250aa44a48f197f681e5d1f34272 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:08.613171 27380 leader_election.cc:302] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:08.613319 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452b250aa44a48f197f681e5d1f34272" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:08.613408 27577 raft_consensus.cc:3037] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.614156 3290 raft_consensus.cc:2781] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:08.614244 3290 raft_consensus.cc:686] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:08.614363 3290 consensus_queue.cc:237] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.614997 27577 raft_consensus.cc:2454] T 452b250aa44a48f197f681e5d1f34272 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:08.617624 24487 catalog_manager.cc:5484] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:08.618458 3293 tablet_bootstrap.cc:492] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:08.618541 3293 ts_tablet_manager.cc:1367] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:59:08.619930 3293 raft_consensus.cc:348] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.620103 3293 raft_consensus.cc:374] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:08.620182 3293 raft_consensus.cc:729] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.620338 3293 consensus_queue.cc:260] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.621779 3293 ts_tablet_manager.cc:1398] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:08.632761 24482 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:59:08.637578 25032 raft_consensus.cc:1261] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:08.637741 27577 raft_consensus.cc:1261] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Refusing update from remote peer 588bfc090a5247598cd5d3cee5ad2007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:08.638020 3235 consensus_queue.cc:1035] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.638222 3067 consensus_queue.cc:1035] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:08.640640 3311 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:08.641019 3311 tablet.cc:1620] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:08.641295 24482 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]). Expected version 1 got 0
I20240502 01:59:08.641716 3312 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:08.642021 3312 tablet.cc:1620] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:08.642872 24487 catalog_manager.cc:5576] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901) does not have the latest schema for tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]). Expected version 1 got 0
I20240502 01:59:08.644047 24482 catalog_manager.cc:5576] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673) does not have the latest schema for tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]). Expected version 1 got 0
I20240502 01:59:08.646239 3313 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:59:08.646536 3313 tablet.cc:1620] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:08.647006 24479 catalog_manager.cc:5576] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879) does not have the latest schema for tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014]). Expected version 1 got 0
I20240502 01:59:08.647236 24482 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014] alter complete (version 1)
I20240502 01:59:08.749089 2738 raft_consensus.cc:482] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:08.749219 2738 raft_consensus.cc:504] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.749552 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:59:08.749712 2738 leader_election.cc:288] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:08.749912 3067 raft_consensus.cc:482] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:08.750054 3067 raft_consensus.cc:504] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.750452 3067 leader_election.cc:288] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:08.751039 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eca9d656f5542d1855e727be59449e1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:08.751212 24897 raft_consensus.cc:2454] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:59:08.751240 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eca9d656f5542d1855e727be59449e1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:08.751350 24763 raft_consensus.cc:2454] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 0.
I20240502 01:59:08.751515 27514 leader_election.cc:302] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb653087e8554c7591cf547dd969f3be, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:08.751053 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8d9037cece45b49c4b0b399e62fd5f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:08.751650 24896 raft_consensus.cc:2454] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:08.752002 2870 raft_consensus.cc:2781] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:08.752064 2870 raft_consensus.cc:482] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:08.752097 2870 raft_consensus.cc:3037] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.752671 24567 leader_election.cc:302] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:08.752888 3067 raft_consensus.cc:2781] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:08.752934 3067 raft_consensus.cc:482] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:08.752965 3067 raft_consensus.cc:3037] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.753204 2870 raft_consensus.cc:504] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.753620 2870 leader_election.cc:288] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 92cb785c165749eca49af5784756461c (127.17.243.2:45237)
I20240502 01:59:08.754319 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8d9037cece45b49c4b0b399e62fd5f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:08.754439 27577 raft_consensus.cc:2454] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:08.755390 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eca9d656f5542d1855e727be59449e1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:08.755518 24763 raft_consensus.cc:3037] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.756017 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eca9d656f5542d1855e727be59449e1" candidate_uuid: "bb653087e8554c7591cf547dd969f3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:08.756115 24896 raft_consensus.cc:3037] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.756197 3311 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:08.756538 3311 tablet.cc:1620] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: MemRowSet was empty: no flush needed.
I20240502 01:59:08.756558 24763 raft_consensus.cc:2454] T 2eca9d656f5542d1855e727be59449e1 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:59:08.757059 24896 raft_consensus.cc:2454] T 2eca9d656f5542d1855e727be59449e1 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb653087e8554c7591cf547dd969f3be in term 1.
I20240502 01:59:08.757057 27514 leader_election.cc:302] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92cb785c165749eca49af5784756461c, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:08.757439 2738 raft_consensus.cc:2781] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:08.757519 2738 raft_consensus.cc:686] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [term 1 LEADER]: Becoming Leader. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Running, Role: LEADER
I20240502 01:59:08.757628 2738 consensus_queue.cc:237] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } }
I20240502 01:59:08.760807 3313 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:08.760926 24479 catalog_manager.cc:5484] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be reported cstate change: term changed from 0 to 1, leader changed from <none> to bb653087e8554c7591cf547dd969f3be (127.17.243.8). New cstate: current_term: 1 leader_uuid: "bb653087e8554c7591cf547dd969f3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:08.761098 3313 tablet.cc:1620] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: MemRowSet was empty: no flush needed.
I20240502 01:59:08.765512 24487 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table [id=09ec24318a3149a395b812cbd27e3014] alter complete (version 2)
I20240502 01:59:08.765908 3312 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:59:08.762982 3067 raft_consensus.cc:504] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.766214 3312 tablet.cc:1620] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: MemRowSet was empty: no flush needed.
I20240502 01:59:08.766435 3067 leader_election.cc:288] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:08.766858 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8d9037cece45b49c4b0b399e62fd5f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:08.766963 24896 raft_consensus.cc:3037] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.767874 24896 raft_consensus.cc:2454] T 1f8d9037cece45b49c4b0b399e62fd5f P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:08.768275 24567 leader_election.cc:302] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:08.769059 3140 raft_consensus.cc:2781] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:08.769145 3140 raft_consensus.cc:686] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:08.769275 3140 consensus_queue.cc:237] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:08.771886 24479 catalog_manager.cc:5484] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } }
I20240502 01:59:08.773609 27577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8d9037cece45b49c4b0b399e62fd5f" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:08.773727 27577 raft_consensus.cc:3037] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:08.774763 27577 raft_consensus.cc:2454] T 1f8d9037cece45b49c4b0b399e62fd5f P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:08.867401 24481 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:59:08.872445 3311 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:08.873270 3313 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:08.876829 3312 tablet.cc:1722] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:59:08.878590 24481 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: rebalancer_dynamic_table_new_name [id=09ec24318a3149a395b812cbd27e3014] alter complete (version 3)
W20240502 01:59:08.899255 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:08.930514 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:08.943401 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:08.980414 24479 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:08.980572 24479 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47628:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:59:08.982697 24479 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 52b027cea09c41dca3dade2a5ec516e9: Sending DeleteTablet for 3 replicas of tablet 3d638889a39b4847960594bdf86079eb
I20240502 01:59:08.983217 27555 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d638889a39b4847960594bdf86079eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:08 UTC) from {username='slave'} at 127.0.0.1:54936
I20240502 01:59:08.983263 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d638889a39b4847960594bdf86079eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:08 UTC) from {username='slave'} at 127.0.0.1:33024
I20240502 01:59:08.983558 3371 tablet_replica.cc:330] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: stopping tablet replica
I20240502 01:59:08.983676 3372 tablet_replica.cc:330] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: stopping tablet replica
I20240502 01:59:08.983709 3371 raft_consensus.cc:2229] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:08.983690 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d638889a39b4847960594bdf86079eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:59:08 UTC) from {username='slave'} at 127.0.0.1:37764
I20240502 01:59:08.983841 3371 raft_consensus.cc:2258] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.983865 3372 raft_consensus.cc:2229] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:08.983971 3373 tablet_replica.cc:330] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: stopping tablet replica
I20240502 01:59:08.984045 3372 raft_consensus.cc:2258] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.984123 3373 raft_consensus.cc:2229] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:59:08.984354 3373 raft_consensus.cc:2258] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.984618 3371 ts_tablet_manager.cc:1875] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:08.984867 3373 ts_tablet_manager.cc:1875] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:08.984906 3372 ts_tablet_manager.cc:1875] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:59:08.986685 3371 ts_tablet_manager.cc:1888] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:08.986786 3371 log.cc:1196] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-7/wal/wals/3d638889a39b4847960594bdf86079eb
I20240502 01:59:08.987247 3372 ts_tablet_manager.cc:1888] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:08.987252 3373 ts_tablet_manager.cc:1888] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:59:08.987435 3373 log.cc:1196] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d638889a39b4847960594bdf86079eb
I20240502 01:59:08.987509 3372 log.cc:1196] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting WAL directory at /tmp/dist-test-task5aC8CU/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714615075709001-18380-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d638889a39b4847960594bdf86079eb
I20240502 01:59:08.987772 3371 ts_tablet_manager.cc:1909] T 3d638889a39b4847960594bdf86079eb P bb653087e8554c7591cf547dd969f3be: Deleting consensus metadata
I20240502 01:59:08.988058 3373 ts_tablet_manager.cc:1909] T 3d638889a39b4847960594bdf86079eb P 588bfc090a5247598cd5d3cee5ad2007: Deleting consensus metadata
I20240502 01:59:08.988058 3372 ts_tablet_manager.cc:1909] T 3d638889a39b4847960594bdf86079eb P e0b9e7df4e8543fa9b8e3ffc24330657: Deleting consensus metadata
I20240502 01:59:08.988550 24474 catalog_manager.cc:4830] TS bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673): tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table_new_name [id=09ec24318a3149a395b812cbd27e3014]) successfully deleted
I20240502 01:59:08.988816 24471 catalog_manager.cc:4830] TS e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879): tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table_new_name [id=09ec24318a3149a395b812cbd27e3014]) successfully deleted
I20240502 01:59:08.989120 24473 catalog_manager.cc:4830] TS 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901): tablet 3d638889a39b4847960594bdf86079eb (table rebalancer_dynamic_table_new_name [id=09ec24318a3149a395b812cbd27e3014]) successfully deleted
W20240502 01:59:09.066135 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:09.067457 3099 consensus_queue.cc:1035] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.070272 3289 consensus_queue.cc:1035] T 452b250aa44a48f197f681e5d1f34272 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:09.084260 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:59:09.084235 24487 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:59:09.089193 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 70fc1d91e9c04674b829248e31a2cb46 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b03461a59c88410e98e01fa9f80b6605]), partition=
I20240502 01:59:09.089399 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 70fc1d91e9c04674b829248e31a2cb46 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b03461a59c88410e98e01fa9f80b6605]), partition=
I20240502 01:59:09.089457 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fc1d91e9c04674b829248e31a2cb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.089653 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fc1d91e9c04674b829248e31a2cb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.091554 3293 tablet_bootstrap.cc:492] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c: Bootstrap starting.
I20240502 01:59:09.091603 3253 tablet_bootstrap.cc:492] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:09.091773 24877 tablet_service.cc:1467] Processing CreateTablet for tablet 70fc1d91e9c04674b829248e31a2cb46 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b03461a59c88410e98e01fa9f80b6605]), partition=
I20240502 01:59:09.091997 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70fc1d91e9c04674b829248e31a2cb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.092315 3253 tablet_bootstrap.cc:654] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.092444 3293 tablet_bootstrap.cc:654] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.093808 3293 tablet_bootstrap.cc:492] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c: No bootstrap required, opened a new log
I20240502 01:59:09.093897 3293 ts_tablet_manager.cc:1367] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:09.094211 3295 tablet_bootstrap.cc:492] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:09.094393 3253 tablet_bootstrap.cc:492] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:09.094374 3293 raft_consensus.cc:348] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.094462 3253 ts_tablet_manager.cc:1367] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:09.094489 3293 raft_consensus.cc:374] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.094527 3293 raft_consensus.cc:729] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cb785c165749eca49af5784756461c, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.094642 3293 consensus_queue.cc:260] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.094921 3293 ts_tablet_manager.cc:1398] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:09.094915 3253 raft_consensus.cc:348] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.095038 3253 raft_consensus.cc:374] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.095069 3253 raft_consensus.cc:729] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.095118 3295 tablet_bootstrap.cc:654] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.095185 3253 consensus_queue.cc:260] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.095453 3253 ts_tablet_manager.cc:1398] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:09.096573 3295 tablet_bootstrap.cc:492] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:09.096653 3295 ts_tablet_manager.cc:1367] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:09.097127 3295 raft_consensus.cc:348] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.097225 3295 raft_consensus.cc:374] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.097256 3295 raft_consensus.cc:729] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.097352 3295 consensus_queue.cc:260] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.097605 3295 ts_tablet_manager.cc:1398] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:09.116552 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:59:09.155571 3339 consensus_queue.cc:1035] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.159643 3339 consensus_queue.cc:1035] T 2eca9d656f5542d1855e727be59449e1 P bb653087e8554c7591cf547dd969f3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.230461 3140 consensus_queue.cc:1035] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.233476 3235 consensus_queue.cc:1035] T 1f8d9037cece45b49c4b0b399e62fd5f P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:09.274900 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:59:09.283699 24479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47628:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:09.290063 25012 tablet_service.cc:1467] Processing CreateTablet for tablet c2e068904d224b70ae51c8083d2002c5 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.290253 27156 tablet_service.cc:1467] Processing CreateTablet for tablet c2e068904d224b70ae51c8083d2002c5 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.290352 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e068904d224b70ae51c8083d2002c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.290495 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e068904d224b70ae51c8083d2002c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.290799 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 200dcdced01d4f4885f1974ad9718363 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.291035 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200dcdced01d4f4885f1974ad9718363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.292860 3253 tablet_bootstrap.cc:492] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:09.292831 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 8718e97e3b3141388371b1cdadff902b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.293059 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8718e97e3b3141388371b1cdadff902b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.293390 3412 tablet_bootstrap.cc:492] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:09.293370 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 200dcdced01d4f4885f1974ad9718363 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.293521 3413 tablet_bootstrap.cc:492] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc: Bootstrap starting.
I20240502 01:59:09.293606 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200dcdced01d4f4885f1974ad9718363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.293839 3253 tablet_bootstrap.cc:654] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.294705 27423 tablet_service.cc:1467] Processing CreateTablet for tablet 200dcdced01d4f4885f1974ad9718363 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.294974 27423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200dcdced01d4f4885f1974ad9718363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.294916 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 8718e97e3b3141388371b1cdadff902b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.295156 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8718e97e3b3141388371b1cdadff902b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.295289 24877 tablet_service.cc:1467] Processing CreateTablet for tablet c2e068904d224b70ae51c8083d2002c5 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.295545 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e068904d224b70ae51c8083d2002c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.297219 3412 tablet_bootstrap.cc:654] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.297595 27555 tablet_service.cc:1467] Processing CreateTablet for tablet 8718e97e3b3141388371b1cdadff902b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=4aaa8a78e31c4c59a65da7832057d217]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:09.297688 3413 tablet_bootstrap.cc:654] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.297883 27555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8718e97e3b3141388371b1cdadff902b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:09.298826 3413 tablet_bootstrap.cc:492] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc: No bootstrap required, opened a new log
I20240502 01:59:09.298897 3413 ts_tablet_manager.cc:1367] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:59:09.299350 3413 raft_consensus.cc:348] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.299445 3413 raft_consensus.cc:374] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.299474 3413 raft_consensus.cc:729] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd619a595e94004ac0f7f5a9054e7cc, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.299579 3413 consensus_queue.cc:260] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.299839 3413 ts_tablet_manager.cc:1398] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:09.300002 3295 tablet_bootstrap.cc:492] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b: Bootstrap starting.
I20240502 01:59:09.300133 3417 tablet_bootstrap.cc:492] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be: Bootstrap starting.
I20240502 01:59:09.300871 3295 tablet_bootstrap.cc:654] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.300952 3417 tablet_bootstrap.cc:654] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.303427 3414 tablet_bootstrap.cc:492] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:09.304585 3417 tablet_bootstrap.cc:492] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be: No bootstrap required, opened a new log
I20240502 01:59:09.304621 3414 tablet_bootstrap.cc:654] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.304674 3417 ts_tablet_manager.cc:1367] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:09.304996 3295 tablet_bootstrap.cc:492] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b: No bootstrap required, opened a new log
I20240502 01:59:09.305071 3295 ts_tablet_manager.cc:1367] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:09.305143 3417 raft_consensus.cc:348] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.305236 3417 raft_consensus.cc:374] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.305263 3417 raft_consensus.cc:729] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb653087e8554c7591cf547dd969f3be, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.305377 3417 consensus_queue.cc:260] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.305487 3295 raft_consensus.cc:348] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.305577 3295 raft_consensus.cc:374] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.305604 3295 raft_consensus.cc:729] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bcd2bc124471981349b1e2726253b, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.305675 3417 ts_tablet_manager.cc:1398] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:09.305742 3295 consensus_queue.cc:260] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.306051 3295 ts_tablet_manager.cc:1398] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:09.306645 3253 tablet_bootstrap.cc:492] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:09.306751 3253 ts_tablet_manager.cc:1367] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:59:09.307181 3414 tablet_bootstrap.cc:492] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:09.307255 3414 ts_tablet_manager.cc:1367] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:09.307214 3253 raft_consensus.cc:348] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.307299 3253 raft_consensus.cc:374] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.307327 3253 raft_consensus.cc:729] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.307438 3253 consensus_queue.cc:260] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.307693 3253 ts_tablet_manager.cc:1398] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:09.307655 3414 raft_consensus.cc:348] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.307739 3414 raft_consensus.cc:374] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.307765 3414 raft_consensus.cc:729] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.307847 3253 tablet_bootstrap.cc:492] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007: Bootstrap starting.
I20240502 01:59:09.307863 3414 consensus_queue.cc:260] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.307899 3412 tablet_bootstrap.cc:492] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:09.307977 3412 ts_tablet_manager.cc:1367] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:59:09.308106 3414 ts_tablet_manager.cc:1398] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:09.308261 3414 tablet_bootstrap.cc:492] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Bootstrap starting.
I20240502 01:59:09.308648 3412 raft_consensus.cc:348] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.308737 3412 raft_consensus.cc:374] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.308764 3412 raft_consensus.cc:729] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.308825 3253 tablet_bootstrap.cc:654] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.308861 3412 consensus_queue.cc:260] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.309144 3412 ts_tablet_manager.cc:1398] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:09.310807 3414 tablet_bootstrap.cc:654] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.312763 3414 tablet_bootstrap.cc:492] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09: No bootstrap required, opened a new log
I20240502 01:59:09.312888 3414 ts_tablet_manager.cc:1367] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:09.314325 3253 tablet_bootstrap.cc:492] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007: No bootstrap required, opened a new log
I20240502 01:59:09.314409 3253 ts_tablet_manager.cc:1367] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:59:09.314801 3253 raft_consensus.cc:348] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.314900 3253 raft_consensus.cc:374] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.314940 3253 raft_consensus.cc:729] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.315052 3253 consensus_queue.cc:260] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.315258 3414 raft_consensus.cc:348] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.315331 3253 ts_tablet_manager.cc:1398] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:09.315348 3414 raft_consensus.cc:374] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.315376 3414 raft_consensus.cc:729] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.315558 3414 consensus_queue.cc:260] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.316406 3414 ts_tablet_manager.cc:1398] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:09.317759 3412 tablet_bootstrap.cc:492] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657: Bootstrap starting.
I20240502 01:59:09.318670 3412 tablet_bootstrap.cc:654] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:09.320645 3412 tablet_bootstrap.cc:492] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657: No bootstrap required, opened a new log
I20240502 01:59:09.320715 3412 ts_tablet_manager.cc:1367] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:09.321537 3412 raft_consensus.cc:348] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.321852 3412 raft_consensus.cc:374] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:09.321913 3412 raft_consensus.cc:729] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Initialized, Role: FOLLOWER
I20240502 01:59:09.322093 3412 consensus_queue.cc:260] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.322374 3412 ts_tablet_manager.cc:1398] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 01:59:09.356364 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:09.381317 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:59:09.409303 3287 raft_consensus.cc:482] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:09.409412 3287 raft_consensus.cc:504] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.409812 3287 leader_election.cc:288] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:09.410336 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e068904d224b70ae51c8083d2002c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
W20240502 01:59:09.410377 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:59:09.410462 24897 raft_consensus.cc:2454] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:09.410472 3140 raft_consensus.cc:482] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:09.410557 3140 raft_consensus.cc:504] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.410969 3140 leader_election.cc:288] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:09.410960 24966 leader_election.cc:302] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0b9e7df4e8543fa9b8e3ffc24330657, f46bcd2bc124471981349b1e2726253b; no voters:
I20240502 01:59:09.411198 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e068904d224b70ae51c8083d2002c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" is_pre_election: true
I20240502 01:59:09.411326 27175 raft_consensus.cc:2454] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 0.
I20240502 01:59:09.411461 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fc1d91e9c04674b829248e31a2cb46" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c" is_pre_election: true
I20240502 01:59:09.411589 24763 raft_consensus.cc:2454] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:09.411770 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fc1d91e9c04674b829248e31a2cb46" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b" is_pre_election: true
I20240502 01:59:09.411883 24897 raft_consensus.cc:2454] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 0.
I20240502 01:59:09.411989 3287 raft_consensus.cc:2781] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:09.412037 3287 raft_consensus.cc:482] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:09.412070 3287 raft_consensus.cc:3037] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.412168 24567 leader_election.cc:302] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:09.412520 3140 raft_consensus.cc:2781] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:09.412575 3140 raft_consensus.cc:482] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:09.412606 3140 raft_consensus.cc:3037] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.413028 3287 raft_consensus.cc:504] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.413353 3287 leader_election.cc:288] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Requested vote from peers f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445), 9dd619a595e94004ac0f7f5a9054e7cc (127.17.243.5:40239)
I20240502 01:59:09.413623 3140 raft_consensus.cc:504] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.413919 3140 leader_election.cc:288] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Requested vote from peers 92cb785c165749eca49af5784756461c (127.17.243.2:45237), f46bcd2bc124471981349b1e2726253b (127.17.243.3:45445)
I20240502 01:59:09.414322 24763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fc1d91e9c04674b829248e31a2cb46" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92cb785c165749eca49af5784756461c"
I20240502 01:59:09.414433 24763 raft_consensus.cc:3037] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.414407 27175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e068904d224b70ae51c8083d2002c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd619a595e94004ac0f7f5a9054e7cc"
I20240502 01:59:09.414500 27175 raft_consensus.cc:3037] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.415285 24763 raft_consensus.cc:2454] T 70fc1d91e9c04674b829248e31a2cb46 P 92cb785c165749eca49af5784756461c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:09.415396 27175 raft_consensus.cc:2454] T c2e068904d224b70ae51c8083d2002c5 P 9dd619a595e94004ac0f7f5a9054e7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:09.415537 24567 leader_election.cc:302] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588bfc090a5247598cd5d3cee5ad2007, 92cb785c165749eca49af5784756461c; no voters:
I20240502 01:59:09.415743 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e068904d224b70ae51c8083d2002c5" candidate_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:09.415822 24965 leader_election.cc:302] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd619a595e94004ac0f7f5a9054e7cc, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:09.415843 24897 raft_consensus.cc:3037] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.415999 3140 raft_consensus.cc:2781] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:09.416085 3140 raft_consensus.cc:686] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 LEADER]: Becoming Leader. State: Replica: 588bfc090a5247598cd5d3cee5ad2007, State: Running, Role: LEADER
I20240502 01:59:09.416208 3140 consensus_queue.cc:237] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } }
I20240502 01:59:09.416810 24897 raft_consensus.cc:2454] T c2e068904d224b70ae51c8083d2002c5 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b9e7df4e8543fa9b8e3ffc24330657 in term 1.
I20240502 01:59:09.416946 3287 raft_consensus.cc:2781] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:09.417040 3287 raft_consensus.cc:686] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9e7df4e8543fa9b8e3ffc24330657, State: Running, Role: LEADER
I20240502 01:59:09.417155 3287 consensus_queue.cc:237] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } }
I20240502 01:59:09.417230 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70fc1d91e9c04674b829248e31a2cb46" candidate_uuid: "588bfc090a5247598cd5d3cee5ad2007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46bcd2bc124471981349b1e2726253b"
I20240502 01:59:09.417318 24897 raft_consensus.cc:3037] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.418120 24897 raft_consensus.cc:2454] T 70fc1d91e9c04674b829248e31a2cb46 P f46bcd2bc124471981349b1e2726253b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588bfc090a5247598cd5d3cee5ad2007 in term 1.
I20240502 01:59:09.419615 24487 catalog_manager.cc:5484] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1). New cstate: current_term: 1 leader_uuid: "588bfc090a5247598cd5d3cee5ad2007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:09.421870 24481 catalog_manager.cc:5484] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4). New cstate: current_term: 1 leader_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:09.424577 3289 raft_consensus.cc:482] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:09.424671 3289 raft_consensus.cc:504] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.425001 3289 leader_election.cc:288] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:09.425318 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200dcdced01d4f4885f1974ad9718363" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:09.425356 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200dcdced01d4f4885f1974ad9718363" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" is_pre_election: true
I20240502 01:59:09.425436 24632 raft_consensus.cc:2454] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:09.425458 25031 raft_consensus.cc:2454] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:09.425810 27377 leader_election.cc:302] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, e0b9e7df4e8543fa9b8e3ffc24330657; no voters:
I20240502 01:59:09.426007 3289 raft_consensus.cc:2781] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:09.426069 3289 raft_consensus.cc:482] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:09.426115 3289 raft_consensus.cc:3037] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.426877 3289 raft_consensus.cc:504] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.427186 3289 leader_election.cc:288] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers e0b9e7df4e8543fa9b8e3ffc24330657 (127.17.243.4:42879), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:09.427564 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200dcdced01d4f4885f1974ad9718363" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:09.427582 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200dcdced01d4f4885f1974ad9718363" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657"
I20240502 01:59:09.427670 24632 raft_consensus.cc:3037] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.427670 25031 raft_consensus.cc:3037] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.428542 25031 raft_consensus.cc:2454] T 200dcdced01d4f4885f1974ad9718363 P e0b9e7df4e8543fa9b8e3ffc24330657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:09.428552 24632 raft_consensus.cc:2454] T 200dcdced01d4f4885f1974ad9718363 P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:09.428792 27379 leader_election.cc:302] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:59:09.428938 3289 raft_consensus.cc:2781] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:09.429021 3289 raft_consensus.cc:686] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:09.429143 3289 consensus_queue.cc:237] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.432842 24481 catalog_manager.cc:5484] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20240502 01:59:09.477841 3289 raft_consensus.cc:482] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:09.477977 3289 raft_consensus.cc:504] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.478339 3289 leader_election.cc:288] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:09.478706 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8718e97e3b3141388371b1cdadff902b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007" is_pre_election: true
I20240502 01:59:09.478709 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8718e97e3b3141388371b1cdadff902b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be" is_pre_election: true
I20240502 01:59:09.478826 24632 raft_consensus.cc:2454] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:09.478847 27578 raft_consensus.cc:2454] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 0.
I20240502 01:59:09.479141 27379 leader_election.cc:302] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, 588bfc090a5247598cd5d3cee5ad2007; no voters:
I20240502 01:59:09.479358 3289 raft_consensus.cc:2781] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:09.479406 3289 raft_consensus.cc:482] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:09.479434 3289 raft_consensus.cc:3037] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.480255 3289 raft_consensus.cc:504] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.480569 3289 leader_election.cc:288] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Requested vote from peers bb653087e8554c7591cf547dd969f3be (127.17.243.8:44673), 588bfc090a5247598cd5d3cee5ad2007 (127.17.243.1:36901)
I20240502 01:59:09.480839 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8718e97e3b3141388371b1cdadff902b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb653087e8554c7591cf547dd969f3be"
I20240502 01:59:09.480965 27578 raft_consensus.cc:3037] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.480957 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8718e97e3b3141388371b1cdadff902b" candidate_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588bfc090a5247598cd5d3cee5ad2007"
I20240502 01:59:09.481077 24632 raft_consensus.cc:3037] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:09.482038 27578 raft_consensus.cc:2454] T 8718e97e3b3141388371b1cdadff902b P bb653087e8554c7591cf547dd969f3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:09.482159 24632 raft_consensus.cc:2454] T 8718e97e3b3141388371b1cdadff902b P 588bfc090a5247598cd5d3cee5ad2007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0a6338e0a467dbdc6efb3e67a6a09 in term 1.
I20240502 01:59:09.482349 27380 leader_election.cc:302] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0a6338e0a467dbdc6efb3e67a6a09, bb653087e8554c7591cf547dd969f3be; no voters:
I20240502 01:59:09.482563 3289 raft_consensus.cc:2781] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:09.482652 3289 raft_consensus.cc:686] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0a6338e0a467dbdc6efb3e67a6a09, State: Running, Role: LEADER
I20240502 01:59:09.482782 3289 consensus_queue.cc:237] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } }
I20240502 01:59:09.485648 24479 catalog_manager.cc:5484] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0a6338e0a467dbdc6efb3e67a6a09 (127.17.243.7). New cstate: current_term: 1 leader_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc0a6338e0a467dbdc6efb3e67a6a09" member_type: VOTER last_known_addr { host: "127.17.243.7" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
W20240502 01:59:09.490762 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:09.522465 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:09.522509 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:09.528739 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:09.604696 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35596116445411f857f52eadd963a16: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:09.722016 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:59:09.840684 3067 consensus_queue.cc:1035] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92cb785c165749eca49af5784756461c" member_type: VOTER last_known_addr { host: "127.17.243.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.846585 3140 consensus_queue.cc:1035] T 70fc1d91e9c04674b829248e31a2cb46 P 588bfc090a5247598cd5d3cee5ad2007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:09.854094 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:09.856587 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:09.896806 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:09.935840 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:59:09.950206 3289 consensus_queue.cc:1035] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b9e7df4e8543fa9b8e3ffc24330657" member_type: VOTER last_known_addr { host: "127.17.243.4" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.953469 3289 consensus_queue.cc:1035] T 200dcdced01d4f4885f1974ad9718363 P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.984086 3290 consensus_queue.cc:1035] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588bfc090a5247598cd5d3cee5ad2007" member_type: VOTER last_known_addr { host: "127.17.243.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.984450 3287 consensus_queue.cc:1035] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd619a595e94004ac0f7f5a9054e7cc" member_type: VOTER last_known_addr { host: "127.17.243.5" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.988430 1891 consensus_queue.cc:1035] T c2e068904d224b70ae51c8083d2002c5 P e0b9e7df4e8543fa9b8e3ffc24330657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bcd2bc124471981349b1e2726253b" member_type: VOTER last_known_addr { host: "127.17.243.3" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:09.990496 3290 consensus_queue.cc:1035] T 8718e97e3b3141388371b1cdadff902b P 1cc0a6338e0a467dbdc6efb3e67a6a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb653087e8554c7591cf547dd969f3be" member_type: VOTER last_known_addr { host: "127.17.243.8" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:10.104435 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.140280 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:59:10.153553 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:10.159440 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.164856 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:59:10.214440 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.227598 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:10.228298 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.256805 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.259542 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:59:10.305733 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.309821 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 01:59:10.343456 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.396518 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.485985 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.492902 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.538311 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:10.540736 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 01:59:10.541409 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.670887 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:10.720548 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.728161 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.912923 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:10.938367 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:59:11.127575 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.204989 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:11.353055 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:11.421015 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.521500 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.651566 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.660501 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:11.672850 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.857036 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.858834 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:11.914824 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.066570 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.085472 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:12.167573 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.231678 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:12.259804 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.294624 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.363247 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.480894 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.544528 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.553531 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:59:12.632863 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.735272 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.762856 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:12.834849 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.839424 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.852147 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.862736 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.933949 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.944489 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:12.960991 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.003055 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.074793 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.189257 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.249023 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.261440 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.300802 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:13.429368 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.604195 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:13.653398 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:13.704685 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:13.952832 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.056921 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.084786 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.124588 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:14.300575 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.372659 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:14.420334 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.470806 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:14.503242 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:14.509475 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.592291 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:14.627878 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:14.658560 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.792205 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:59:14.837460 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:14.949725 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.020005 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:15.078249 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.095065 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.190460 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 01:59:15.252089 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 01:59:15.294726 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:15.295346 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.306914 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.357698 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.366128 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.369815 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.407156 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.408136 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.445576 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.454375 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:15.478490 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.543612 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:15.593227 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 01:59:15.594272 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:15.615470 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.645917 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.668623 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.684064 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.696502 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.852149 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:15.976452 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:16.296223 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:16.299669 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:16.323601 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:16.329418 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:59:16.330031 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:16.399942 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:16.504921 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:16.795900 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:16.870954 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:16.874804 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:16.898438 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:16.916517 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:17.106424 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.133652 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.161589 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:17.213003 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.314332 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:17.354431 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.448767 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:59:17.558262 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.578446 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.730244 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.781271 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.793334 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.794509 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.839149 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.841734 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.849748 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.874918 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:17.889382 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:17.997718 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.027151 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.124697 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.124992 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.134868 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.137449 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.191284 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.198580 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.450131 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:18.678759 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.864635 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.920253 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:18.921834 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:18.978673 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.127920 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:19.313887 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:19.413265 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:19.434731 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:19.453892 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.508545 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:19.560207 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.630990 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:19.732393 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.748207 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.794116 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.838357 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:19.901018 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:19.990526 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.015625 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:59:20.032577 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 01:59:20.033180 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet 252502ad2e124996b345888706e674d5: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:20.033237 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 252502ad2e124996b345888706e674d5 on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:20.033879 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 7db1384cb0374899a6e17a550e1f7cba on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:20.113530 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.268332 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.271080 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.274296 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:59:20.276674 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:59:20.277222 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.290756 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:20.345342 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.390481 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.407135 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.417191 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.440385 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:20.440984 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.535513 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.537366 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:20.537894 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.560257 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:20.598255 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.616080 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.652369 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 01:59:20.652961 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.679844 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.717885 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 6e608c8945504b18b1cd82d421f2d3b9 on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:20.718071 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet a2d5afdc9ff6458ca834de042f8e14ab on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:20.718719 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 1d9ee234307a410ca578be50f798804d on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:20.866945 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:20.913492 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:21.124976 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:21.172736 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:21.344715 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:21.389997 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:21.570703 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:21.603572 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:21.604668 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): CreateTablet RPC failed for tablet da60bdbf6831421e9f9abec53c6e1d35: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:21.604770 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet da60bdbf6831421e9f9abec53c6e1d35 on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:21.604899 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 57e64ca6504646b8b77c5d36e32d5d6c on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:21.738019 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:21.844154 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:21.864687 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:21.965147 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:21.965796 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:21.997746 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:22.055869 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:22.066668 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:22.252215 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.276881 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.345562 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.371943 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:22.400894 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.429158 24471 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 0180449195154cb49ebccb8b604af4e2 on TS 1049b4c8783b48f39be22bd113ad349f
W20240502 01:59:22.469761 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.504634 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.544246 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:59:22.572124 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.596060 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.607285 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.622180 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.736318 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.751679 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.807304 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:22.911424 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:22.959710 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.041851 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.070003 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.084389 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.088522 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.093255 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.147750 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b2686f7b58341e9a0b1f2b27cfbb173: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:23.161729 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.284096 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.386689 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:23.796116 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:23.802958 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.842617 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.977468 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:23.978422 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:24.050411 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:24.193998 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.247824 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:24.335723 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.426100 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:24.436475 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:24.446238 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.527294 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:24.569398 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.790107 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.798009 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.805513 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:24.841550 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.864146 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.919811 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:24.951395 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.044035 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.070991 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.101131 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.158653 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:59:25.178581 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.209408 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.292299 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 01:59:25.293289 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 01:59:25.293593 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.298223 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 01:59:25.299171 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 781ba9bbf19e413bb8981c48073f3360: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:25.310813 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:25.351917 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.375864 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.425503 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.460959 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:59:25.568673 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.630790 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:25.691361 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.705659 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:25.706418 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.709443 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.807595 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:25.849539 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 01:59:25.856268 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:26.129101 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:26.150321 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:26.385334 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:26.432317 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:26.484206 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:26.488448 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:26.581756 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:26.662675 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:59:26.978085 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.000008 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.000020 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:27.000319 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:27.028342 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.118105 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.121063 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:27.132556 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.165975 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:27.212486 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.225842 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.331416 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.362288 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.394532 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.429172 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:27.440138 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:59:27.458938 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.491618 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.583750 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.628993 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.643850 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.736121 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.783641 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:27.863216 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:27.952021 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.071698 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.127125 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.191100 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.367444 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.468710 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:28.482156 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.574594 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.617885 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:28.769464 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:29.046940 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:29.070698 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:29.128962 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:29.216087 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.328673 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:29.379236 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.437763 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.466596 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.517606 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.533178 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:29.535567 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:29.545612 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:29.779196 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.822619 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.827337 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.830895 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.845947 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.859071 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.920446 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:29.940042 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.017449 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.023377 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.072973 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:30.129824 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.135071 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.218410 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.295747 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:59:30.336802 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 01:59:30.343977 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 01:59:30.421823 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.440822 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.449137 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.655376 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:30.759721 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:59:30.760403 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.802790 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:30.803313 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.832015 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:30.893695 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:30.952656 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 01:59:31.152992 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:31.231884 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:31.279887 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:31.323239 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:31.526182 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:31.611022 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:31.652307 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:31.819828 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:31.835371 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:31.888116 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:31.906852 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.026793 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:32.064747 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.088002 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:32.110224 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.127007 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:32.128927 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.143177 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.354350 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.399089 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.429209 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.449365 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.473572 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.479338 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.499569 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.519924 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.530798 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.560526 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.568248 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.624753 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.636564 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:32.666756 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:32.962318 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:33.027314 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:59:33.175453 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:33.211212 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:33.306295 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:33.442297 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:33.555224 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:33.658833 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:33.669247 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:33.670331 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.032672 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.056058 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.096542 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.115453 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:34.239465 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.264989 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.300026 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.356024 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.434823 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.529793 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:34.553766 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.618954 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.660778 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.705413 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.705897 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:34.876590 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.879864 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.947469 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.954963 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.959069 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:34.997004 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.001333 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.036428 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.048425 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.052889 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.127722 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.185650 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:35.199985 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.386401 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:59:35.428366 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 01:59:35.484766 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:59:35.500808 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.671590 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.730991 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:35.744412 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:35.811462 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:59:35.842586 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:35.922964 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:35.931933 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.100196 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 01:59:36.147845 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.173228 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:36.343369 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.389513 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.432415 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.705794 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:36.710891 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.740259 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.757750 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:36.817623 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.865320 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:36.913130 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:36.973986 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.004649 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:37.038513 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.225690 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.246073 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:37.279840 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.298920 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.304904 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:37.349977 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.389796 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.410251 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.469686 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.480949 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.492161 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.538776 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.569245 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.686870 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.793864 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.839809 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.845890 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:37.892021 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:37.975291 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.059687 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.108937 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:59:38.209847 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.314450 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.419916 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:38.549712 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.627377 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.815049 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:38.963619 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:39.135437 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:39.163100 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:39.192747 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:39.211673 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:39.214313 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:39.268703 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.347209 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.357313 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:39.583645 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.627247 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.644063 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:39.660141 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.722296 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.776367 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.788067 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.951234 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.957304 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:39.977128 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:39.990909 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.015640 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.034468 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.085633 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.111407 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.430891 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:40.446005 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:59:40.460029 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.515739 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:59:40.529693 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.623652 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.637980 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:40.842268 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:40.845949 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:40.849727 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:59:40.862083 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.118666 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:41.134186 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 01:59:41.134922 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.134999 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.135039 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.161054 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 01:59:41.196305 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.349108 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:41.414079 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.448029 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.671795 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.672915 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.714803 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.787150 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.815799 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:41.824674 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:41.958130 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.060756 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:42.177063 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.207329 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.289772 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:42.301896 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.357991 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.386830 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:42.420917 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.439116 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.474740 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.608217 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.626058 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.687122 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.707494 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.851959 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.898204 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.956465 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:42.960377 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:42.968504 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.066638 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.115550 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:59:43.240803 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.320982 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.393520 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:43.453073 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.504062 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.605581 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.666517 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.734980 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.758720 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:43.768779 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:44.012504 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.118603 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.217206 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.232671 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:44.238454 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.261837 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.404376 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.492089 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:44.518517 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:44.650521 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:44.662092 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:44.665135 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:44.722491 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:44.837138 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:44.966655 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:44.996122 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.074247 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:45.084554 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.170289 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.192487 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.308084 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.315639 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.415032 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.452638 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 01:59:45.453260 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.454360 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.457518 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.461686 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.551960 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.564664 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:59:45.698963 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:45.915409 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:59:45.915971 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:45.923197 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:45.951013 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.078531 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.079186 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.096966 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.175786 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240502 01:59:46.176340 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.195226 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 01:59:46.195744 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:46.267779 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.441651 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.490099 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.507413 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:59:46.721338 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.751021 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:46.782068 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.806388 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.921254 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:46.944131 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:46.984443 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.188529 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.290839 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:47.301693 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.347700 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.371318 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:47.386245 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.472003 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.541328 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:47.714774 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:47.723765 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.738297 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.742900 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.774263 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.804272 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.820633 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.830674 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.899127 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.907536 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:59:47.964792 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:47.968030 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.055313 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.114547 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:48.383329 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.517318 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.527309 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.571489 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.596379 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:48.618155 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.641687 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:48.649823 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.042367 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.088045 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:49.103891 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.172979 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.246381 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.247169 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.298852 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.383172 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.526877 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.562076 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:49.575280 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:49.776275 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.858474 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.864363 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.871325 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:49.951553 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:49.953303 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:50.016908 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.191818 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.195073 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.222191 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.292366 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.350445 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.390252 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:50.404273 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.422595 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.585527 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.608924 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.649304 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:59:50.789834 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:50.799655 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 01:59:50.812839 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.825830 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.904103 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.926187 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.931900 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:59:50.943405 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.946573 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:50.983645 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:51.009383 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:51.203969 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 01:59:51.235466 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 01:59:51.356925 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:51.372846 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:51.471818 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:51.596138 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:51.639192 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:51.679054 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:51.715256 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:51.777911 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:51.786617 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:59:51.911688 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:51.991250 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:52.020381 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.107648 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.121011 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:52.201030 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.209184 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:52.212000 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.286082 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.534308 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.579001 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.588862 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:52.733583 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.735549 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.746074 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:52.763103 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:59:52.875780 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.885962 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.887641 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.947566 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.955003 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:52.982878 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:59:53.064077 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.203459 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:53.257537 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.277320 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.393203 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.422773 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.534775 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.661295 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.709815 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:53.759992 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:54.061448 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.086100 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.119033 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.143860 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.170624 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.224579 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.360599 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:54.428340 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.559973 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.577951 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.605270 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.618527 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.721885 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.746390 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.769464 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:54.798818 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:54.837123 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:54.978960 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:55.075204 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.094852 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.208761 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.253655 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:55.257864 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.322741 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.408280 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.448259 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.480316 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:59:55.548789 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.665808 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.763964 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.848426 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:55.854640 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 01:59:55.854719 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:55.941540 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 01:59:55.942253 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b2686f7b58341e9a0b1f2b27cfbb173: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:55.972627 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.013324 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:56.055594 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.061239 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.068797 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:56.105204 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.262822 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:59:56.278934 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.349710 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 01:59:56.376685 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.405064 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:56.494307 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.586303 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:56.670167 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.703866 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.780920 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:56.846022 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:56.941761 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:56.964658 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:56.981278 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:56.982200 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:56.992935 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:56.999706 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.030347 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.115213 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:57.219834 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.457720 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:57.467914 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:57.493279 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:57.537016 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.548007 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.672390 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.779556 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:57.838908 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.919945 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:57.923853 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:57.956552 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:59:58.036407 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.077155 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 781ba9bbf19e413bb8981c48073f3360: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:58.235025 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.238780 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.252430 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.282483 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.339159 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.407073 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:58.486384 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.517849 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.581238 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.655105 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.715045 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.842339 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.844466 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:58.898650 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:59.021607 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:59.077260 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:59.083204 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.106776 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:59.386782 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.428555 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.435801 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.458622 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.470753 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.530445 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.646150 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 01:59:59.711093 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:59:59.727571 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:59:59.900085 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.063613 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:00.088436 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.121577 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:00.168362 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.186846 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:00.226424 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.251471 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:00.378909 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.480525 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.658118 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:00.701422 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.813311 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.896718 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:00:00.897367 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.914893 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.935518 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:00.953087 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.002508 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.005710 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.048584 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:00:01.050796 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.106560 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.186471 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:01.213187 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:00:01.239396 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.265393 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:00:01.351234 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.425033 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:01.431746 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.446269 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.471381 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:01.683079 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:01.788888 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:01.797302 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:01.799682 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:01.905344 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.979626 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:01.992060 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.068728 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.146047 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.206512 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:02.228603 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.372463 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:02.407126 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.451537 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.490818 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.512334 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:02.614753 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:02.671944 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.688711 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:02.735841 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:02.776744 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:02.838522 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:00:03.024435 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.266320 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.313503 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.316877 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:03.344214 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.400826 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.421123 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.439383 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.595316 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.618369 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.691756 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.788358 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.791409 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.793105 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.847211 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.880270 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:03.918145 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:03.918390 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:04.126860 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.314288 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.408253 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.468410 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:04.500054 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:04.559794 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.678889 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.705909 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:04.735097 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.779331 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.862107 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:04.871726 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.874938 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:04.939587 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.063046 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.078773 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:05.154928 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:05.156342 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:05.341792 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.615363 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.661628 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:05.678578 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.712077 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.716215 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.888173 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:05.986866 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:00:06.089629 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:06.113662 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 02:00:06.132723 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.146714 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.219283 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.233628 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.294373 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.297469 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 02:00:06.298039 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.337827 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:06.338376 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.368589 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.442024 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.492877 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:06.585247 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:06.602483 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:06.632853 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 02:00:06.633484 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:06.698000 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:06.874024 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.027053 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.075536 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:07.148837 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.154224 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.192932 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.220613 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.233362 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:07.251297 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:07.305954 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:07.339200 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.339308 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.351006 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:07.415042 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.624742 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:07.626569 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:07.673061 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:07.847532 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:07.857285 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:07.976051 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.116617 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.141875 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:08.208250 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.233091 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.241570 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.514696 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.610740 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.671721 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.691327 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.712023 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:08.732652 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.749130 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:08.756083 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.913992 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:08.952786 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:09.021956 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:09.198709 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.214569 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:09.223073 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:09.339200 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.397025 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.488224 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.491032 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.610512 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:09.641917 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:09.686138 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.735225 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.736296 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.803733 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:09.915454 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.940076 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:09.974874 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:09.992507 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:10.092783 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:10.151084 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:10.254535 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:10.332315 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:00:10.406890 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:10.591164 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:10.596424 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:10.834443 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:10.859784 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.001236 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:00:11.005489 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:11.008358 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.064589 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.099846 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.099889 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.102964 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.124812 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:11.149802 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:11.308017 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.317890 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:00:11.367867 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:00:11.368422 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.396528 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.445977 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:11.528230 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.544675 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:11.596673 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.637683 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:11.691985 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:11.891901 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:11.894093 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:11.906893 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:11.958880 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:12.056636 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:12.071897 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.136286 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.140884 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:12.258540 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.261673 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.369490 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:12.379405 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.380335 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:12.437141 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:12.466912 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.492496 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.500602 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:12.847996 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:12.902552 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:12.916642 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:00:12.951550 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:12.959311 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.057240 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.216272 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.338976 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.358973 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.379032 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:13.519351 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:13.547493 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.592877 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.614125 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.656898 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.718132 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.816680 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.911098 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.912897 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:13.994009 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:14.010244 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:14.103268 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:14.106838 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.261436 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:14.265673 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.270447 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.502476 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.562433 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.568246 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.634553 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:14.768469 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:14.819722 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.845798 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:14.932649 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:14.954921 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.045339 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:15.109119 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.119194 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.166119 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.190313 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:15.385785 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:00:15.459017 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:15.509148 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.637979 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.778698 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.808437 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.816313 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:15.829008 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.926828 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:15.966704 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:15.973263 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.044440 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:00:16.152698 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.183956 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:16.222651 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.287427 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.334826 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:00:16.368104 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:16.368666 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.422585 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.457089 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.488082 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:16.529222 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:16.556810 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:16.707628 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.726004 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:16.751735 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:16.871484 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:16.884145 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:16.921978 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:17.016845 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.067633 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:17.117815 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.256458 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.265898 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:17.415920 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.416301 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.429744 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:17.517730 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.519075 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:17.520535 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:17.545102 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.594967 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.653327 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:17.801638 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:17.952385 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:18.006714 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:00:18.182507 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.196204 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.334021 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:18.373565 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.466017 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:18.470242 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.529382 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.539068 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:18.728230 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.732164 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.761183 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.766074 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.878182 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.945780 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:18.985862 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:19.006227 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:19.025017 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.047219 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:19.107429 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:19.115298 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.224364 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.238392 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.367611 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.446231 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:19.455178 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:19.747565 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:19.834969 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.836123 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.948915 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:19.951035 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:20.030035 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.070009 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:20.102205 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.126013 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.180516 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.260149 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.363135 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:20.563584 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.630328 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:20.669409 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:00:20.745782 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.850011 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:20.903467 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.057240 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.078318 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 02:00:21.078974 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:21.088270 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.145812 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.202370 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:21.270959 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.300915 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:00:21.350430 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:00:21.351302 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.467443 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.531663 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:21.532289 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.543545 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:21.626269 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:21.649506 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.688022 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.689191 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:21.705945 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:21.717278 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:21.769728 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.006093 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:22.019506 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.064401 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:22.114760 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:22.170495 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.198562 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.255468 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:00:22.313472 24476 maintenance_manager.cc:419] P 52b027cea09c41dca3dade2a5ec516e9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033366
I20240502 02:00:22.346220 24475 maintenance_manager.cc:643] P 52b027cea09c41dca3dade2a5ec516e9: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.028s sys 0.004s Metrics: {"bytes_written":113484,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":315,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":158}
I20240502 02:00:22.347086 24476 maintenance_manager.cc:419] P 52b027cea09c41dca3dade2a5ec516e9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 41056 bytes on disk
I20240502 02:00:22.347607 24475 maintenance_manager.cc:643] P 52b027cea09c41dca3dade2a5ec516e9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240502 02:00:22.390578 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.510902 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:22.537992 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.554790 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:22.613574 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:22.627475 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.687206 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.735949 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:22.772563 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:23.030663 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.144502 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:23.154403 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:23.157351 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:00:23.167608 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.224051 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.334895 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:23.335517 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.412240 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:23.455529 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.583257 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:23.633798 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.772996 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:23.780933 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.797467 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.808899 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.928660 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:23.974694 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.037215 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.049654 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.076416 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.080631 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.082698 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:24.215689 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.381600 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.516422 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.537709 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:24.604343 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.621701 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.733551 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.812163 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:24.862285 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.927557 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:24.964895 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:24.980402 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.002303 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.010651 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.395033 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.468881 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:25.472672 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:25.483103 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:00:25.531299 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.560576 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.620410 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.712421 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:25.830499 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:25.846109 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:25.979251 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.107075 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:00:26.126940 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.135337 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.225224 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:26.334647 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.406864 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:00:26.407456 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:26.413017 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.416536 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.441605 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:26.449329 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:26.456650 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.457312 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.459496 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.535107 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:26.565093 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.608300 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.694918 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:26.802886 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:26.923616 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:27.026640 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.084522 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.138547 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:27.178680 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.229537 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.261993 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:00:27.415196 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.448989 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.454833 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.457705 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.465782 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:27.497607 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:27.665306 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:27.834599 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.877058 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:27.890761 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:27.917233 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:00:27.975915 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:00:28.005275 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(26422e12ddf64e27b3913c60800e5a56): perf score=0.033336
I20240502 02:00:28.031245 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(26422e12ddf64e27b3913c60800e5a56) complete. Timing: real 0.026s user 0.024s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":676,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1054,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":380,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":291,"threads_started":3,"wal-append.queue_time_us":322}
I20240502 02:00:28.035001 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(26422e12ddf64e27b3913c60800e5a56): 4579 bytes on disk
I20240502 02:00:28.035669 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(26422e12ddf64e27b3913c60800e5a56) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240502 02:00:28.071388 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:28.079018 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:28.095417 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(48085cebf94e4f549e6174a88acdcc59): perf score=0.033361
I20240502 02:00:28.099840 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(44be529dec974eac8419efe9c19b30bf): perf score=0.033364
I20240502 02:00:28.139078 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(48085cebf94e4f549e6174a88acdcc59) complete. Timing: real 0.043s user 0.027s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":171,"dirs.queue_time_us":7764,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":290,"threads_started":3,"wal-append.queue_time_us":347}
I20240502 02:00:28.141469 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(48085cebf94e4f549e6174a88acdcc59): 4921 bytes on disk
I20240502 02:00:28.142644 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(48085cebf94e4f549e6174a88acdcc59) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 02:00:28.144423 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(26422e12ddf64e27b3913c60800e5a56): perf score=0.033374
W20240502 02:00:28.175451 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:00:28.175539 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(48085cebf94e4f549e6174a88acdcc59): perf score=0.033386
W20240502 02:00:28.185149 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:00:28.207754 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(26422e12ddf64e27b3913c60800e5a56) complete. Timing: real 0.063s user 0.016s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1306,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":42508}
I20240502 02:00:28.209756 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(26422e12ddf64e27b3913c60800e5a56): 4579 bytes on disk
I20240502 02:00:28.210224 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(26422e12ddf64e27b3913c60800e5a56) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 02:00:28.201733 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(44be529dec974eac8419efe9c19b30bf) complete. Timing: real 0.102s user 0.028s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":303,"dirs.queue_time_us":7130,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":4083,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":251,"threads_started":3,"wal-append.queue_time_us":8391}
I20240502 02:00:28.215749 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(44be529dec974eac8419efe9c19b30bf): 4834 bytes on disk
I20240502 02:00:28.218456 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(44be529dec974eac8419efe9c19b30bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 02:00:28.259183 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(48085cebf94e4f549e6174a88acdcc59) complete. Timing: real 0.083s user 0.033s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":4738,"dirs.queue_time_us":7217,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":7492,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":228,"threads_started":3,"wal-append.queue_time_us":10355}
I20240502 02:00:28.261848 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(48085cebf94e4f549e6174a88acdcc59): 4921 bytes on disk
I20240502 02:00:28.262562 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(48085cebf94e4f549e6174a88acdcc59) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 02:00:28.301653 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:28.647101 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:28.657464 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:28.658918 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:00:28.690166 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:28.838882 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:28.841585 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:28.870186 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:28.944635 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.036250 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:29.041837 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.118927 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:29.159041 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:29.194761 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:00:29.218711 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(488816b0d5d542bcb5fa12ff7e70fd40): perf score=0.033371
W20240502 02:00:29.224906 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:29.226205 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(49d6b2dc45b84ed99ef017b736db25a5): perf score=0.033375
I20240502 02:00:29.233434 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(488816b0d5d542bcb5fa12ff7e70fd40) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1425,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":133}
I20240502 02:00:29.235266 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(488816b0d5d542bcb5fa12ff7e70fd40): 4834 bytes on disk
I20240502 02:00:29.235662 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(488816b0d5d542bcb5fa12ff7e70fd40) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 02:00:29.241241 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:00:29.244396 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(49d6b2dc45b84ed99ef017b736db25a5) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":424,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":123}
I20240502 02:00:29.245914 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(49d6b2dc45b84ed99ef017b736db25a5): 4922 bytes on disk
I20240502 02:00:29.246289 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(49d6b2dc45b84ed99ef017b736db25a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:29.271227 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(49d6b2dc45b84ed99ef017b736db25a5): perf score=0.033387
I20240502 02:00:29.284590 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(49d6b2dc45b84ed99ef017b736db25a5) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1532,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":400,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":138}
I20240502 02:00:29.286139 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(49d6b2dc45b84ed99ef017b736db25a5): 4922 bytes on disk
I20240502 02:00:29.286504 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(49d6b2dc45b84ed99ef017b736db25a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 02:00:29.288066 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(b3edec06b3d8450a8cc3002be70c848a): perf score=0.033389
I20240502 02:00:29.297086 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(49d6b2dc45b84ed99ef017b736db25a5): perf score=0.033395
W20240502 02:00:29.304694 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:00:29.304944 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(b3edec06b3d8450a8cc3002be70c848a) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1514,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":909,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":153}
I20240502 02:00:29.306901 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(b3edec06b3d8450a8cc3002be70c848a): 4581 bytes on disk
I20240502 02:00:29.307315 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(b3edec06b3d8450a8cc3002be70c848a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 02:00:29.318755 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(49d6b2dc45b84ed99ef017b736db25a5) complete. Timing: real 0.021s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":176}
I20240502 02:00:29.320506 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(49d6b2dc45b84ed99ef017b736db25a5): 4922 bytes on disk
I20240502 02:00:29.320885 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(49d6b2dc45b84ed99ef017b736db25a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240502 02:00:29.347903 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.388888 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.454376 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:29.636662 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:29.767763 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.776657 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.793257 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.890967 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:29.931764 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.964252 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:29.981904 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.208279 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:30.209697 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:30.239275 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:30.243917 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(8d5f6bcbe0a048eb85b9475348b155ef): perf score=0.033359
I20240502 02:00:30.254236 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(70d0c3a7075a4e40a621b73931a3a33b): perf score=0.033363
I20240502 02:00:30.261405 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(8d5f6bcbe0a048eb85b9475348b155ef) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":309,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1497,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":309}
I20240502 02:00:30.262938 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(8d5f6bcbe0a048eb85b9475348b155ef): 4581 bytes on disk
I20240502 02:00:30.263314 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(8d5f6bcbe0a048eb85b9475348b155ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 02:00:30.267889 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(70d0c3a7075a4e40a621b73931a3a33b) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1083,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":511,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":153}
I20240502 02:00:30.269785 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(70d0c3a7075a4e40a621b73931a3a33b): 4834 bytes on disk
I20240502 02:00:30.270167 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(70d0c3a7075a4e40a621b73931a3a33b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240502 02:00:30.314720 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(26b7501639064035a570b3b4c1cf083f): perf score=0.033380
I20240502 02:00:30.328887 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(70d0c3a7075a4e40a621b73931a3a33b): perf score=0.033382
I20240502 02:00:30.330758 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(26b7501639064035a570b3b4c1cf083f) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":368}
I20240502 02:00:30.340370 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(8d5f6bcbe0a048eb85b9475348b155ef): perf score=0.033387
I20240502 02:00:30.345292 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(70d0c3a7075a4e40a621b73931a3a33b) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":351,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1195,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":144}
I20240502 02:00:30.347110 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(70d0c3a7075a4e40a621b73931a3a33b): 4834 bytes on disk
I20240502 02:00:30.347432 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(70d0c3a7075a4e40a621b73931a3a33b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240502 02:00:30.356324 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(8d5f6bcbe0a048eb85b9475348b155ef) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1163,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1348,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":182}
I20240502 02:00:30.357811 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(26b7501639064035a570b3b4c1cf083f): 4921 bytes on disk
I20240502 02:00:30.358225 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(26b7501639064035a570b3b4c1cf083f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 02:00:30.360044 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(8d5f6bcbe0a048eb85b9475348b155ef): 4581 bytes on disk
I20240502 02:00:30.360400 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(8d5f6bcbe0a048eb85b9475348b155ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240502 02:00:30.374722 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:30.520903 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.545620 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.555176 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.576496 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.638995 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:30.648056 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.660867 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:30.692276 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:00:31.057255 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.092540 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.150274 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:31.153995 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:00:31.188570 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:31.236515 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:00:31.270788 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(92382869887b43f9a23a2c0ea0012720): perf score=0.033343
I20240502 02:00:31.278682 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(4ac6a4310330499483182c93827343c0): perf score=0.033346
W20240502 02:00:31.278818 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:00:31.293505 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(92382869887b43f9a23a2c0ea0012720) complete. Timing: real 0.023s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1430,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":8064,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":1429}
I20240502 02:00:31.293505 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(4ac6a4310330499483182c93827343c0) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3365,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1404,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":213}
I20240502 02:00:31.295439 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(4ac6a4310330499483182c93827343c0): 4921 bytes on disk
I20240502 02:00:31.295837 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(92382869887b43f9a23a2c0ea0012720): 4834 bytes on disk
I20240502 02:00:31.296099 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(4ac6a4310330499483182c93827343c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 02:00:31.297667 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(69182e5f4ae7401f89d22f063fa8df63): perf score=0.033350
I20240502 02:00:31.298463 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(92382869887b43f9a23a2c0ea0012720) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 02:00:31.315532 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(69182e5f4ae7401f89d22f063fa8df63) complete. Timing: real 0.017s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1228,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":385,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":5103}
I20240502 02:00:31.317308 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(69182e5f4ae7401f89d22f063fa8df63): 4580 bytes on disk
I20240502 02:00:31.317694 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(69182e5f4ae7401f89d22f063fa8df63) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 02:00:31.355090 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(4ac6a4310330499483182c93827343c0): perf score=0.033368
I20240502 02:00:31.368127 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(92382869887b43f9a23a2c0ea0012720): perf score=0.033369
I20240502 02:00:31.368973 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(4ac6a4310330499483182c93827343c0) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1503,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":413,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":140}
I20240502 02:00:31.370699 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(4ac6a4310330499483182c93827343c0): 4921 bytes on disk
I20240502 02:00:31.371356 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(4ac6a4310330499483182c93827343c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 02:00:31.382154 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(92382869887b43f9a23a2c0ea0012720) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":139,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":427,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":117}
I20240502 02:00:31.383631 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(92382869887b43f9a23a2c0ea0012720): 4834 bytes on disk
I20240502 02:00:31.383955 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(92382869887b43f9a23a2c0ea0012720) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240502 02:00:31.388262 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.473910 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:00:31.474881 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:31.503679 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:31.640027 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:31.675797 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:31.676513 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.684357 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.695600 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:31.710201 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.727974 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:31.769007 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:32.011524 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:32.176497 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.196442 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:32.196918 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:32.248852 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.312964 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.332422 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.409130 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:32.413597 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.418061 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.505743 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:32.506381 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:00:32.558161 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(89580549c13d4c1ebdb2a5d24da2a01e): perf score=0.033359
I20240502 02:00:32.576278 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(89580549c13d4c1ebdb2a5d24da2a01e) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1402,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":530,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":131}
I20240502 02:00:32.577478 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(e720d8363bb049938798ab0b9aabc28d): perf score=0.033366
I20240502 02:00:32.578151 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(938581084b27405cbc01917e1a982208): perf score=0.033363
W20240502 02:00:32.582924 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:00:32.599283 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(938581084b27405cbc01917e1a982208) complete. Timing: real 0.021s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2730,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":5223,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":474,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":5354}
I20240502 02:00:32.600860 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(89580549c13d4c1ebdb2a5d24da2a01e): 4921 bytes on disk
I20240502 02:00:32.601161 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(89580549c13d4c1ebdb2a5d24da2a01e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240502 02:00:32.601687 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(e720d8363bb049938798ab0b9aabc28d) complete. Timing: real 0.024s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1132,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":133}
W20240502 02:00:32.602348 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:32.602840 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(938581084b27405cbc01917e1a982208): 4581 bytes on disk
I20240502 02:00:32.603168 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(938581084b27405cbc01917e1a982208) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 02:00:32.603475 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(e720d8363bb049938798ab0b9aabc28d): 4834 bytes on disk
I20240502 02:00:32.604005 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(e720d8363bb049938798ab0b9aabc28d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":186,"lbm_reads_lt_1ms":4}
W20240502 02:00:32.609417 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:00:32.630043 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(89580549c13d4c1ebdb2a5d24da2a01e): perf score=0.033382
I20240502 02:00:32.642513 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(938581084b27405cbc01917e1a982208): perf score=0.033384
I20240502 02:00:32.643791 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(89580549c13d4c1ebdb2a5d24da2a01e) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1347,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":132}
I20240502 02:00:32.645200 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(89580549c13d4c1ebdb2a5d24da2a01e): 4921 bytes on disk
I20240502 02:00:32.645491 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(89580549c13d4c1ebdb2a5d24da2a01e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 02:00:32.655813 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(938581084b27405cbc01917e1a982208) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":130,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1474,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":429}
I20240502 02:00:32.657171 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(938581084b27405cbc01917e1a982208): 4581 bytes on disk
I20240502 02:00:32.657474 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(938581084b27405cbc01917e1a982208) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240502 02:00:32.855283 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:32.958567 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:32.966379 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:00:32.967100 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.008358 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.043789 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.197758 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:33.218152 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.304906 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.346750 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:00:33.380447 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.511747 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.542166 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:00:33.611312 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(9b8667a49f1a44339122040b0122be0d): perf score=0.033345
I20240502 02:00:33.611837 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(953a778a807a484b9e0376bdeeed8343): perf score=0.033347
W20240502 02:00:33.620193 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:00:33.624126 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(9b8667a49f1a44339122040b0122be0d) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":336,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1469,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":111}
I20240502 02:00:33.626129 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(9b8667a49f1a44339122040b0122be0d): 4581 bytes on disk
I20240502 02:00:33.626284 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(953a778a807a484b9e0376bdeeed8343) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1204,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":428,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":108,"threads_started":2,"wal-append.queue_time_us":142}
I20240502 02:00:33.626616 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(9b8667a49f1a44339122040b0122be0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 02:00:33.627844 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(953a778a807a484b9e0376bdeeed8343): 4921 bytes on disk
I20240502 02:00:33.628059 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(086c9b674a0543cab3c82349d151e373): perf score=0.033349
I20240502 02:00:33.628229 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(953a778a807a484b9e0376bdeeed8343) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 02:00:33.642165 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(086c9b674a0543cab3c82349d151e373) complete. Timing: real 0.014s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1235,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":210}
I20240502 02:00:33.643819 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(086c9b674a0543cab3c82349d151e373): 4834 bytes on disk
I20240502 02:00:33.644165 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(086c9b674a0543cab3c82349d151e373) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 02:00:33.654064 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(086c9b674a0543cab3c82349d151e373): perf score=0.033356
I20240502 02:00:33.665526 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(9b8667a49f1a44339122040b0122be0d): perf score=0.033363
I20240502 02:00:33.668588 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(086c9b674a0543cab3c82349d151e373) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":2598,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":417}
I20240502 02:00:33.670333 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(086c9b674a0543cab3c82349d151e373): 4834 bytes on disk
I20240502 02:00:33.670751 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(086c9b674a0543cab3c82349d151e373) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 02:00:33.678683 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(9b8667a49f1a44339122040b0122be0d) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 02:00:33.680080 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(953a778a807a484b9e0376bdeeed8343): perf score=0.033365
I20240502 02:00:33.692188 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(953a778a807a484b9e0376bdeeed8343) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":454,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":148}
I20240502 02:00:33.693657 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(953a778a807a484b9e0376bdeeed8343): 4921 bytes on disk
I20240502 02:00:33.694007 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(953a778a807a484b9e0376bdeeed8343) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:33.695576 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(9b8667a49f1a44339122040b0122be0d): 4581 bytes on disk
I20240502 02:00:33.695961 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(9b8667a49f1a44339122040b0122be0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 02:00:33.696300 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:33.840714 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:33.944084 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:33.961611 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:33.962765 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:33.979916 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.041913 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:34.112712 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.135041 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:34.153081 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:34.226651 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.331740 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.570830 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:34.570920 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:34.586977 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.611042 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:34.611390 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:34.629956 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.688400 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:34.842897 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:34.883311 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:00:34.886766 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(645305c993a5486ebc1e6c8b81941128): perf score=0.033366
W20240502 02:00:34.887646 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:00:34.903146 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(645305c993a5486ebc1e6c8b81941128) complete. Timing: real 0.016s user 0.012s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":300,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1627,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":142}
I20240502 02:00:34.904580 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(ebe8db639839433d967583371351eacb): perf score=0.033370
I20240502 02:00:34.905090 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(645305c993a5486ebc1e6c8b81941128): 4834 bytes on disk
I20240502 02:00:34.905465 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(645305c993a5486ebc1e6c8b81941128) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 02:00:34.919152 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(ebe8db639839433d967583371351eacb) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":289,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":143}
I20240502 02:00:34.920503 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(ebe8db639839433d967583371351eacb): 4581 bytes on disk
I20240502 02:00:34.920843 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(ebe8db639839433d967583371351eacb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:34.930235 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(63312ec15b034e47bc2dd484b57d4648): perf score=0.033378
I20240502 02:00:34.944223 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(63312ec15b034e47bc2dd484b57d4648) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 02:00:34.945699 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(63312ec15b034e47bc2dd484b57d4648): 4919 bytes on disk
I20240502 02:00:34.946082 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(63312ec15b034e47bc2dd484b57d4648) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 02:00:34.947361 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(ebe8db639839433d967583371351eacb): perf score=0.033381
I20240502 02:00:34.955730 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(645305c993a5486ebc1e6c8b81941128): perf score=0.033381
I20240502 02:00:34.958593 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(ebe8db639839433d967583371351eacb) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":146}
I20240502 02:00:34.959986 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(ebe8db639839433d967583371351eacb): 4581 bytes on disk
I20240502 02:00:34.960300 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(ebe8db639839433d967583371351eacb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:34.968609 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(645305c993a5486ebc1e6c8b81941128) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":135,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1349,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":160}
I20240502 02:00:34.970969 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(645305c993a5486ebc1e6c8b81941128): 4834 bytes on disk
I20240502 02:00:34.971359 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(645305c993a5486ebc1e6c8b81941128) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 02:00:35.077308 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:00:35.088904 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.117148 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:35.250218 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.332008 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.347774 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.405820 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.507820 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:35.558259 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.673122 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.730538 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:35.763842 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.782143 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.814884 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:35.825469 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:00:35.912839 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(ce35d9953e294b4986ff9e93e12db6f6): perf score=0.033335
I20240502 02:00:35.926682 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(ce35d9953e294b4986ff9e93e12db6f6) complete. Timing: real 0.014s user 0.008s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":2014,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":123}
I20240502 02:00:35.928148 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(ce35d9953e294b4986ff9e93e12db6f6): perf score=0.033339
I20240502 02:00:35.928571 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(ce35d9953e294b4986ff9e93e12db6f6): 4581 bytes on disk
I20240502 02:00:35.929131 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(ce35d9953e294b4986ff9e93e12db6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 02:00:35.941145 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(ce35d9953e294b4986ff9e93e12db6f6) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2669,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":456,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":273,"threads_started":2,"wal-append.queue_time_us":195}
I20240502 02:00:35.942975 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(ce35d9953e294b4986ff9e93e12db6f6): 4581 bytes on disk
I20240502 02:00:35.943532 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(ce35d9953e294b4986ff9e93e12db6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240502 02:00:35.943727 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:00:35.967689 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(0e47667fc8fa49e6b18ee3df06cda00a): perf score=0.033350
I20240502 02:00:35.978971 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(0e47667fc8fa49e6b18ee3df06cda00a): perf score=0.033356
I20240502 02:00:35.984653 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(0e47667fc8fa49e6b18ee3df06cda00a) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":2436,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":186}
I20240502 02:00:35.986553 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(91d0d9525b5f4384861dc50d0bf4967d): perf score=0.033351
I20240502 02:00:35.996805 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(0e47667fc8fa49e6b18ee3df06cda00a) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1244,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":151}
I20240502 02:00:35.998811 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(0e47667fc8fa49e6b18ee3df06cda00a): 4921 bytes on disk
I20240502 02:00:35.999280 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(0e47667fc8fa49e6b18ee3df06cda00a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:36.004849 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(91d0d9525b5f4384861dc50d0bf4967d) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":356}
I20240502 02:00:36.006592 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(0e47667fc8fa49e6b18ee3df06cda00a): 4921 bytes on disk
I20240502 02:00:36.007622 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(0e47667fc8fa49e6b18ee3df06cda00a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 02:00:36.009073 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(91d0d9525b5f4384861dc50d0bf4967d): 4834 bytes on disk
I20240502 02:00:36.009465 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(91d0d9525b5f4384861dc50d0bf4967d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240502 02:00:36.161034 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:00:36.173743 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:36.228490 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:36.321236 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:36.330802 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:36.497290 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:36.505999 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:00:36.506595 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:36.528272 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 02:00:36.528924 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:36.550330 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:36.680209 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:36.805959 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:00:36.806571 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:36.807708 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:36.809188 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:36.925814 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.001078 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.001955 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:37.117638 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.134099 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:00:37.188849 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(93695e2165e142f79293ac14e4f4fb84): perf score=0.033375
I20240502 02:00:37.203783 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(d136e006dd1048739156526af42fddd1): perf score=0.033377
I20240502 02:00:37.206050 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(93695e2165e142f79293ac14e4f4fb84) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":148}
I20240502 02:00:37.207566 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(93695e2165e142f79293ac14e4f4fb84): 4834 bytes on disk
I20240502 02:00:37.208060 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(93695e2165e142f79293ac14e4f4fb84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 02:00:37.217900 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(d136e006dd1048739156526af42fddd1) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":226,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1413,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":377,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":113,"threads_started":2,"wal-append.queue_time_us":116}
I20240502 02:00:37.219871 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(d136e006dd1048739156526af42fddd1): 4581 bytes on disk
I20240502 02:00:37.220232 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(d136e006dd1048739156526af42fddd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 02:00:37.245775 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:00:37.258683 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(7d0cc9aba3804968a6dce97682f6b38e): perf score=0.033396
I20240502 02:00:37.269714 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(7d0cc9aba3804968a6dce97682f6b38e): perf score=0.033396
I20240502 02:00:37.272114 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(7d0cc9aba3804968a6dce97682f6b38e) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":158}
I20240502 02:00:37.273692 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(7d0cc9aba3804968a6dce97682f6b38e): 4921 bytes on disk
I20240502 02:00:37.274099 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(7d0cc9aba3804968a6dce97682f6b38e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 02:00:37.283398 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(7d0cc9aba3804968a6dce97682f6b38e) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1337,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":139}
I20240502 02:00:37.284889 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(7d0cc9aba3804968a6dce97682f6b38e): 4921 bytes on disk
I20240502 02:00:37.285223 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(7d0cc9aba3804968a6dce97682f6b38e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 02:00:37.286571 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(d136e006dd1048739156526af42fddd1): perf score=0.033398
W20240502 02:00:37.300154 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:00:37.308682 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(d136e006dd1048739156526af42fddd1) complete. Timing: real 0.022s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1816,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":1392,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":7169}
I20240502 02:00:37.310528 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(d136e006dd1048739156526af42fddd1): 4581 bytes on disk
I20240502 02:00:37.310891 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(d136e006dd1048739156526af42fddd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240502 02:00:37.403836 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:37.414888 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.458635 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:37.499715 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.512609 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:37.714802 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.754935 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:00:37.887614 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:37.892522 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:37.909745 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.026839 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.066056 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.110697 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.143604 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.151335 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:38.237635 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:00:38.282096 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling FlushMRSOp(16ae91433f10465c922183298a387caa): perf score=0.033341
W20240502 02:00:38.294684 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:00:38.298822 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: FlushMRSOp(16ae91433f10465c922183298a387caa) complete. Timing: real 0.017s user 0.012s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3941,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1563,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":445,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":2871}
I20240502 02:00:38.300606 24946 maintenance_manager.cc:419] P f46bcd2bc124471981349b1e2726253b: Scheduling UndoDeltaBlockGCOp(16ae91433f10465c922183298a387caa): 4921 bytes on disk
I20240502 02:00:38.300961 24834 maintenance_manager.cc:643] P f46bcd2bc124471981349b1e2726253b: UndoDeltaBlockGCOp(16ae91433f10465c922183298a387caa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 02:00:38.319093 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(d5ec1158a7414039b6f4bc855207c938): perf score=0.033354
I20240502 02:00:38.332206 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(d5ec1158a7414039b6f4bc855207c938) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":123,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1436,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":392,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":151}
I20240502 02:00:38.334141 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(d5ec1158a7414039b6f4bc855207c938): 4834 bytes on disk
I20240502 02:00:38.334570 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(d5ec1158a7414039b6f4bc855207c938) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 02:00:38.335916 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:38.337117 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling FlushMRSOp(16ae91433f10465c922183298a387caa): perf score=0.033358
W20240502 02:00:38.347275 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:00:38.351181 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: FlushMRSOp(16ae91433f10465c922183298a387caa) complete. Timing: real 0.014s user 0.008s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":132}
I20240502 02:00:38.352553 24812 maintenance_manager.cc:419] P 92cb785c165749eca49af5784756461c: Scheduling UndoDeltaBlockGCOp(16ae91433f10465c922183298a387caa): 4921 bytes on disk
I20240502 02:00:38.352828 24702 maintenance_manager.cc:643] P 92cb785c165749eca49af5784756461c: UndoDeltaBlockGCOp(16ae91433f10465c922183298a387caa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240502 02:00:38.449718 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:00:38.467016 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling FlushMRSOp(61ad1a07bc7a4fbcb901792d138e2843): perf score=0.033392
I20240502 02:00:38.478965 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling FlushMRSOp(61ad1a07bc7a4fbcb901792d138e2843): perf score=0.033397
W20240502 02:00:38.485109 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:00:38.487320 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: FlushMRSOp(61ad1a07bc7a4fbcb901792d138e2843) complete. Timing: real 0.020s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1465,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":2376}
I20240502 02:00:38.489182 24680 maintenance_manager.cc:419] P 588bfc090a5247598cd5d3cee5ad2007: Scheduling UndoDeltaBlockGCOp(61ad1a07bc7a4fbcb901792d138e2843): 4581 bytes on disk
I20240502 02:00:38.489533 24568 maintenance_manager.cc:643] P 588bfc090a5247598cd5d3cee5ad2007: UndoDeltaBlockGCOp(61ad1a07bc7a4fbcb901792d138e2843) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:38.491761 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: FlushMRSOp(61ad1a07bc7a4fbcb901792d138e2843) complete. Timing: real 0.013s user 0.009s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2700,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":451,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":123}
I20240502 02:00:38.493096 25081 maintenance_manager.cc:419] P e0b9e7df4e8543fa9b8e3ffc24330657: Scheduling UndoDeltaBlockGCOp(61ad1a07bc7a4fbcb901792d138e2843): 4581 bytes on disk
I20240502 02:00:38.493393 24967 maintenance_manager.cc:643] P e0b9e7df4e8543fa9b8e3ffc24330657: UndoDeltaBlockGCOp(61ad1a07bc7a4fbcb901792d138e2843) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240502 02:00:38.509325 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:38.537374 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.798861 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.868008 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:38.879755 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:38.977768 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:38.982115 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.082507 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.170557 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.214651 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:00:39.215212 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.221423 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.349094 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.458837 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:39.472878 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.666745 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.685089 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.699606 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.736658 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.808243 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.865038 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.883949 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:39.901096 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:39.993184 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:40.067044 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.186462 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.370867 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:40.382249 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.421862 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.422881 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.577471 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:40.608132 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.610616 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.622812 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:40.675314 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.886381 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:00:40.908124 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.987747 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:40.993376 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:40.999539 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.054376 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:41.257525 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:00:41.268646 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(ce35d9953e294b4986ff9e93e12db6f6): perf score=0.033355
I20240502 02:00:41.292356 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(ce35d9953e294b4986ff9e93e12db6f6) complete. Timing: real 0.023s user 0.022s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":152,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1094,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":13312,"thread_start_us":202,"threads_started":3,"wal-append.queue_time_us":124}
I20240502 02:00:41.294553 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(ce35d9953e294b4986ff9e93e12db6f6): 4581 bytes on disk
I20240502 02:00:41.295171 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(ce35d9953e294b4986ff9e93e12db6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 02:00:41.296912 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(70d0c3a7075a4e40a621b73931a3a33b): perf score=0.033339
W20240502 02:00:41.298936 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:41.315395 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(70d0c3a7075a4e40a621b73931a3a33b) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1680,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":154}
I20240502 02:00:41.316922 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(70d0c3a7075a4e40a621b73931a3a33b): 4834 bytes on disk
I20240502 02:00:41.317386 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(70d0c3a7075a4e40a621b73931a3a33b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 02:00:41.364084 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.488183 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.512706 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:41.548488 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.549525 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 02:00:41.596344 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(b3edec06b3d8450a8cc3002be70c848a): perf score=0.033386
W20240502 02:00:41.601969 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.605257 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:41.633720 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(b3edec06b3d8450a8cc3002be70c848a) complete. Timing: real 0.037s user 0.026s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":153,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1077,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":234,"threads_started":3,"wal-append.queue_time_us":162}
I20240502 02:00:41.636494 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(b3edec06b3d8450a8cc3002be70c848a): 4581 bytes on disk
I20240502 02:00:41.637065 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(b3edec06b3d8450a8cc3002be70c848a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 02:00:41.638590 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(91d0d9525b5f4384861dc50d0bf4967d): perf score=0.033378
I20240502 02:00:41.647322 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(e720d8363bb049938798ab0b9aabc28d): perf score=0.033346
I20240502 02:00:41.659360 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(91d0d9525b5f4384861dc50d0bf4967d) complete. Timing: real 0.021s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1213,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":528,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":375}
I20240502 02:00:41.661655 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(91d0d9525b5f4384861dc50d0bf4967d): 4834 bytes on disk
I20240502 02:00:41.662012 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(91d0d9525b5f4384861dc50d0bf4967d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 02:00:41.688123 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(e720d8363bb049938798ab0b9aabc28d) complete. Timing: real 0.040s user 0.025s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":135,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":752,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":199,"threads_started":3,"wal-append.queue_time_us":139}
I20240502 02:00:41.690327 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(e720d8363bb049938798ab0b9aabc28d): 4834 bytes on disk
I20240502 02:00:41.691269 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(e720d8363bb049938798ab0b9aabc28d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 02:00:41.693074 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(26422e12ddf64e27b3913c60800e5a56): perf score=0.033344
W20240502 02:00:41.697503 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.711544 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:41.719727 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(26422e12ddf64e27b3913c60800e5a56) complete. Timing: real 0.026s user 0.017s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1316,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":106,"threads_started":1,"wal-append.queue_time_us":6760}
I20240502 02:00:41.721782 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(26422e12ddf64e27b3913c60800e5a56): 4579 bytes on disk
I20240502 02:00:41.722183 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(26422e12ddf64e27b3913c60800e5a56) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240502 02:00:41.777285 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:41.812395 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:41.862291 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:00:41.915863 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(69182e5f4ae7401f89d22f063fa8df63): perf score=0.033389
I20240502 02:00:41.931697 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(69182e5f4ae7401f89d22f063fa8df63) complete. Timing: real 0.016s user 0.011s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":468,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":154}
I20240502 02:00:41.933765 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(69182e5f4ae7401f89d22f063fa8df63): 4580 bytes on disk
I20240502 02:00:41.934494 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(69182e5f4ae7401f89d22f063fa8df63) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 02:00:41.965194 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:41.986234 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:42.036053 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.055991 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.118742 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.141650 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.184340 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.212615 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:42.229878 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:42.323662 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:42.420241 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:42.586098 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.705462 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.725446 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:42.766482 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.860262 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.892697 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.931156 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:00:42.944367 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(d5ec1158a7414039b6f4bc855207c938): perf score=0.033366
W20240502 02:00:42.950484 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:42.957585 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(d5ec1158a7414039b6f4bc855207c938) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1355,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":454,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":134}
I20240502 02:00:42.960158 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(d5ec1158a7414039b6f4bc855207c938): 4834 bytes on disk
I20240502 02:00:42.960599 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(d5ec1158a7414039b6f4bc855207c938) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240502 02:00:42.963249 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:42.970916 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:43.034911 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:43.060585 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:43.076449 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:43.083350 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(488816b0d5d542bcb5fa12ff7e70fd40): perf score=0.033403
I20240502 02:00:43.098623 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(488816b0d5d542bcb5fa12ff7e70fd40) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1233,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":132}
I20240502 02:00:43.100435 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(488816b0d5d542bcb5fa12ff7e70fd40): 4834 bytes on disk
I20240502 02:00:43.100803 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(488816b0d5d542bcb5fa12ff7e70fd40) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 02:00:43.102412 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(4ac6a4310330499483182c93827343c0): perf score=0.033366
I20240502 02:00:43.115883 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(4ac6a4310330499483182c93827343c0) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1323,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":198}
I20240502 02:00:43.117702 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(4ac6a4310330499483182c93827343c0): 4921 bytes on disk
I20240502 02:00:43.118062 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(4ac6a4310330499483182c93827343c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 02:00:43.234905 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(63312ec15b034e47bc2dd484b57d4648): perf score=0.033367
I20240502 02:00:43.251950 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(63312ec15b034e47bc2dd484b57d4648) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":983,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1323,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":817}
I20240502 02:00:43.254542 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(63312ec15b034e47bc2dd484b57d4648): 4919 bytes on disk
I20240502 02:00:43.255012 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(63312ec15b034e47bc2dd484b57d4648) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240502 02:00:43.261797 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:43.371598 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(ebe8db639839433d967583371351eacb): perf score=0.033389
I20240502 02:00:43.384325 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(ebe8db639839433d967583371351eacb) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1328,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":198}
I20240502 02:00:43.386075 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(ebe8db639839433d967583371351eacb): 4581 bytes on disk
I20240502 02:00:43.386412 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(ebe8db639839433d967583371351eacb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 02:00:43.421378 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:43.424765 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:00:43.472227 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:43.522058 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:43.611374 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:43.719132 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(9b8667a49f1a44339122040b0122be0d): perf score=0.033361
I20240502 02:00:43.735831 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(9b8667a49f1a44339122040b0122be0d) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":705,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":202,"threads_started":2,"wal-append.queue_time_us":162}
I20240502 02:00:43.739516 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(645305c993a5486ebc1e6c8b81941128): perf score=0.033361
W20240502 02:00:43.744834 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:43.757594 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(645305c993a5486ebc1e6c8b81941128) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":121,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1655,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":150}
I20240502 02:00:43.759873 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(8d5f6bcbe0a048eb85b9475348b155ef): perf score=0.033341
I20240502 02:00:43.776111 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(8d5f6bcbe0a048eb85b9475348b155ef) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":2086,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":212}
I20240502 02:00:43.778390 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(645305c993a5486ebc1e6c8b81941128): 4834 bytes on disk
I20240502 02:00:43.778757 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(645305c993a5486ebc1e6c8b81941128) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 02:00:43.780606 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(8d5f6bcbe0a048eb85b9475348b155ef): 4581 bytes on disk
I20240502 02:00:43.780930 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(8d5f6bcbe0a048eb85b9475348b155ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:00:43.782503 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(9b8667a49f1a44339122040b0122be0d): 4581 bytes on disk
I20240502 02:00:43.782784 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(9b8667a49f1a44339122040b0122be0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4}
W20240502 02:00:43.801622 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:43.891448 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(e720d8363bb049938798ab0b9aabc28d): perf score=0.033380
I20240502 02:00:43.907922 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(e720d8363bb049938798ab0b9aabc28d) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":3469,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":144}
I20240502 02:00:43.910988 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(086c9b674a0543cab3c82349d151e373): perf score=0.033371
I20240502 02:00:43.927114 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(086c9b674a0543cab3c82349d151e373) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1291,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":886,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":122,"threads_started":1,"wal-append.queue_time_us":213}
I20240502 02:00:43.930151 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(086c9b674a0543cab3c82349d151e373): 4834 bytes on disk
I20240502 02:00:43.930681 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(086c9b674a0543cab3c82349d151e373) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240502 02:00:43.931474 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:43.932317 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:43.933981 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(e720d8363bb049938798ab0b9aabc28d): 4834 bytes on disk
I20240502 02:00:43.934382 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(e720d8363bb049938798ab0b9aabc28d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 02:00:44.095141 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:44.152993 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.212484 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:44.238499 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:44.249827 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:44.271414 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.317265 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:44.507051 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.523561 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.568205 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:44.581030 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.590921 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:00:44.642946 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:44.658495 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.756386 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:44.769151 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(488816b0d5d542bcb5fa12ff7e70fd40): perf score=0.033368
I20240502 02:00:44.783224 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(488816b0d5d542bcb5fa12ff7e70fd40) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":2243,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":134}
I20240502 02:00:44.784904 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(488816b0d5d542bcb5fa12ff7e70fd40): 4834 bytes on disk
I20240502 02:00:44.785235 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(488816b0d5d542bcb5fa12ff7e70fd40) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 02:00:44.788748 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:44.863899 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:44.946533 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(44be529dec974eac8419efe9c19b30bf): perf score=0.033337
I20240502 02:00:44.963591 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(44be529dec974eac8419efe9c19b30bf) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1345,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":638,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":143}
I20240502 02:00:44.965591 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(44be529dec974eac8419efe9c19b30bf): 4834 bytes on disk
I20240502 02:00:44.966060 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(44be529dec974eac8419efe9c19b30bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240502 02:00:44.973351 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:45.039011 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(26b7501639064035a570b3b4c1cf083f): perf score=0.033394
I20240502 02:00:45.052727 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(26b7501639064035a570b3b4c1cf083f) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":150}
I20240502 02:00:45.054638 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(26b7501639064035a570b3b4c1cf083f): 4921 bytes on disk
I20240502 02:00:45.055079 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(26b7501639064035a570b3b4c1cf083f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 02:00:45.077667 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.083962 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.180039 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.273005 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.290400 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:00:45.295802 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(89580549c13d4c1ebdb2a5d24da2a01e): perf score=0.033401
I20240502 02:00:45.309502 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(d136e006dd1048739156526af42fddd1): perf score=0.033345
I20240502 02:00:45.312136 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(89580549c13d4c1ebdb2a5d24da2a01e) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2982,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1349,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":1709}
I20240502 02:00:45.314548 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(89580549c13d4c1ebdb2a5d24da2a01e): 4921 bytes on disk
I20240502 02:00:45.314913 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(89580549c13d4c1ebdb2a5d24da2a01e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 02:00:45.320344 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:45.325142 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(d136e006dd1048739156526af42fddd1) complete. Timing: real 0.015s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1156,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":487,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":261}
I20240502 02:00:45.327142 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(d136e006dd1048739156526af42fddd1): 4581 bytes on disk
I20240502 02:00:45.327481 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(d136e006dd1048739156526af42fddd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240502 02:00:45.382077 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:45.384358 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.442955 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.486358 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:45.535112 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:45.569029 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(93695e2165e142f79293ac14e4f4fb84): perf score=0.033398
W20240502 02:00:45.591730 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:45.599341 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(93695e2165e142f79293ac14e4f4fb84) complete. Timing: real 0.030s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1441,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":837,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":11096}
I20240502 02:00:45.602763 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(93695e2165e142f79293ac14e4f4fb84): 4834 bytes on disk
I20240502 02:00:45.603289 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(93695e2165e142f79293ac14e4f4fb84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240502 02:00:45.612892 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:45.802373 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:45.930850 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:46.006280 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:00:46.038379 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.201258 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:46.302438 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:00:46.334122 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:46.337893 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(953a778a807a484b9e0376bdeeed8343): perf score=0.033351
W20240502 02:00:46.340235 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:00:46.351212 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(953a778a807a484b9e0376bdeeed8343) complete. Timing: real 0.013s user 0.009s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":244,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":474,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":184,"threads_started":2,"wal-append.queue_time_us":153}
I20240502 02:00:46.352738 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(953a778a807a484b9e0376bdeeed8343): 4921 bytes on disk
I20240502 02:00:46.353190 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(953a778a807a484b9e0376bdeeed8343) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240502 02:00:46.480921 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.522895 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:00:46.523473 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.554993 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.566324 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:46.567234 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240502 02:00:46.567826 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.707965 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.830579 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:46.858351 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(61ad1a07bc7a4fbcb901792d138e2843): perf score=0.033350
I20240502 02:00:46.871237 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(61ad1a07bc7a4fbcb901792d138e2843) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":507,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":2944,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":140}
I20240502 02:00:46.873050 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(61ad1a07bc7a4fbcb901792d138e2843): 4581 bytes on disk
I20240502 02:00:46.873428 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(61ad1a07bc7a4fbcb901792d138e2843) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 02:00:46.887141 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:46.925308 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:46.948144 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:46.972208 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.018144 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.162007 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:47.181242 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.212810 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.325663 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:47.351030 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.383002 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.403216 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.406457 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.420975 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.532392 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.651674 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:47.679414 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.722385 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:47.727916 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.779979 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.865173 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:47.935742 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:47.936465 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:48.057747 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:48.186720 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:48.281241 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:48.356897 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
I20240502 02:00:48.384474 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(91d0d9525b5f4384861dc50d0bf4967d): perf score=0.033385
W20240502 02:00:48.390556 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:00:48.401639 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(91d0d9525b5f4384861dc50d0bf4967d) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":978,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1192,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":105,"threads_started":2,"wal-append.queue_time_us":126}
I20240502 02:00:48.403020 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling FlushMRSOp(69182e5f4ae7401f89d22f063fa8df63): perf score=0.033357
I20240502 02:00:48.414484 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: FlushMRSOp(69182e5f4ae7401f89d22f063fa8df63) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1074,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":133}
I20240502 02:00:48.416029 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(91d0d9525b5f4384861dc50d0bf4967d): 4834 bytes on disk
I20240502 02:00:48.416312 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(91d0d9525b5f4384861dc50d0bf4967d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240502 02:00:48.417837 27226 maintenance_manager.cc:419] P 9dd619a595e94004ac0f7f5a9054e7cc: Scheduling UndoDeltaBlockGCOp(69182e5f4ae7401f89d22f063fa8df63): 4580 bytes on disk
I20240502 02:00:48.418219 27114 maintenance_manager.cc:643] P 9dd619a595e94004ac0f7f5a9054e7cc: UndoDeltaBlockGCOp(69182e5f4ae7401f89d22f063fa8df63) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240502 02:00:48.495329 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:48.499975 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:48.621829 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:48.643025 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:48.760536 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:00:48.791847 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:48.849357 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:48.854915 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:48.985567 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:00:48.999795 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(63312ec15b034e47bc2dd484b57d4648): perf score=0.033349
I20240502 02:00:49.015093 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(63312ec15b034e47bc2dd484b57d4648) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1455,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":138}
I20240502 02:00:49.017325 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(63312ec15b034e47bc2dd484b57d4648): 4919 bytes on disk
I20240502 02:00:49.017740 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(63312ec15b034e47bc2dd484b57d4648) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240502 02:00:49.028141 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:00:49.131392 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling FlushMRSOp(44be529dec974eac8419efe9c19b30bf): perf score=0.033361
I20240502 02:00:49.145128 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: FlushMRSOp(44be529dec974eac8419efe9c19b30bf) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1836,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":472,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":122}
I20240502 02:00:49.146844 27493 maintenance_manager.cc:419] P 1cc0a6338e0a467dbdc6efb3e67a6a09: Scheduling UndoDeltaBlockGCOp(44be529dec974eac8419efe9c19b30bf): 4834 bytes on disk
I20240502 02:00:49.147204 27381 maintenance_manager.cc:643] P 1cc0a6338e0a467dbdc6efb3e67a6a09: UndoDeltaBlockGCOp(44be529dec974eac8419efe9c19b30bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240502 02:00:49.154412 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:49.172925 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.348224 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:49.379173 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.392184 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:49.405884 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.430663 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.493362 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:49.550458 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:49.655802 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.823020 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:49.851516 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:49.927769 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.948163 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:49.974658 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.064479 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.076810 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:50.131520 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:50.137902 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.185962 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.251585 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.302917 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:50.421672 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.571731 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.612954 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.616005 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:50.726485 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:50.763840 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:50.879209 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:50.900031 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:00:51.082654 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.164367 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.186579 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:51.191031 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.194366 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.259837 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:00:51.284802 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(d5ec1158a7414039b6f4bc855207c938): perf score=0.033369
I20240502 02:00:51.298064 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(d5ec1158a7414039b6f4bc855207c938) complete. Timing: real 0.013s user 0.011s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1482,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":113,"threads_started":2,"wal-append.queue_time_us":243}
I20240502 02:00:51.300197 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(d5ec1158a7414039b6f4bc855207c938): 4834 bytes on disk
I20240502 02:00:51.300590 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(d5ec1158a7414039b6f4bc855207c938) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 02:00:51.311642 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:00:51.370157 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:51.527761 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:51.579023 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:00:51.659145 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:51.659721 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.686100 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:51.716689 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:51.733428 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:51.778000 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:51.780620 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:51.817839 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.901098 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:51.938042 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:51.958473 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.116847 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:52.180132 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:52.218107 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.327693 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:52.368150 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.419450 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:52.524230 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.531540 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.587558 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:52.648226 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.694018 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.703553 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:52.792660 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:52.881799 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.019862 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:53.124616 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.128443 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:53.171914 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.212853 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.220177 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:53.428438 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.435854 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:53.448081 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:00:53.550655 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:53.623562 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.681650 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.727757 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:53.768653 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.798252 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.865483 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:53.999459 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.052721 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.058671 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.112819 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.197530 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.209235 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.270435 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:54.367316 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:54.428822 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.478551 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:54.514832 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:54.620976 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:54.766903 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:00:54.826857 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling FlushMRSOp(93695e2165e142f79293ac14e4f4fb84): perf score=0.033402
I20240502 02:00:54.843529 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: FlushMRSOp(93695e2165e142f79293ac14e4f4fb84) complete. Timing: real 0.016s user 0.012s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":127,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":2482,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":492,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":115,"threads_started":2,"wal-append.queue_time_us":121}
I20240502 02:00:54.845693 27626 maintenance_manager.cc:419] P bb653087e8554c7591cf547dd969f3be: Scheduling UndoDeltaBlockGCOp(93695e2165e142f79293ac14e4f4fb84): 4834 bytes on disk
I20240502 02:00:54.846092 27515 maintenance_manager.cc:643] P bb653087e8554c7591cf547dd969f3be: UndoDeltaBlockGCOp(93695e2165e142f79293ac14e4f4fb84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240502 02:00:54.849071 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:54.873963 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:54.939292 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.954308 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:54.982007 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.038017 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:55.107342 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:55.168269 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.249264 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.318604 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.368994 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.411530 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.421367 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:55.517251 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.716073 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.722923 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:55.763996 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:55.800798 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:55.870651 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:00:55.878242 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:55.978253 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 02:00:55.978962 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c97e5f78b03245cc86f70b1d50fcf723: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:55.995606 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:56.030608 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:56.195089 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:56.207867 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:56.294018 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:56.324599 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240502 02:00:56.327757 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:56.418948 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.478408 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.522413 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.625241 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.695703 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 02:00:56.696532 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.764376 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:56.807567 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 02:00:56.821425 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.868769 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:56.878583 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:56.953989 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:57.008234 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:57.159019 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:57.161275 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:57.175743 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.225857 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:57.228724 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.287791 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.352435 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.404500 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:57.521526 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.666780 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:57.745213 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:57.932125 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.965210 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:57.969091 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.000766 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.088387 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 781ba9bbf19e413bb8981c48073f3360: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:58.157970 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:58.158867 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:58.161227 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.223575 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.236495 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.244793 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:58.407150 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:58.431758 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.564006 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:58.568218 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.606673 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.622171 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:00:58.683559 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:58.730908 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:58.765607 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:58.907817 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:58.985378 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.063441 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.064903 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.213837 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:59.276033 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:59.440506 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:59.520395 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.631549 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.639158 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:59.657027 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.665681 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:59.674603 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:59.713569 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:00:59.829720 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.875201 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.880360 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:00:59.914742 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:00:59.995088 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.111570 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.118742 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:00.128849 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:00.364696 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:00.387459 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.479614 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.704669 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.719913 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.721240 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.733337 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:00.813709 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.858168 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:00.872745 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:00.964671 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:00.964838 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:01.014945 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:01.111670 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:01.129271 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:01.267596 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:01.413710 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:01.459208 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:01.497506 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:01.529649 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:01:01.565212 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:01.622363 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:01.795305 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:01:01.816684 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:01:01.910804 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:01.974918 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:02.076916 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:02.135217 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:02.144225 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.193475 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:01:02.202208 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:02.218617 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.236891 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.265695 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:02.299705 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.302479 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:02.314250 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.361186 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:02.494554 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.571094 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.576058 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:02.576191 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.745786 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:02.785550 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.903503 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:02.922479 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:03.077698 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:03.103014 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:03.120296 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:03.121690 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:03.133692 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:03.197338 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:03.300228 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:03.341135 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:01:03.580989 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:03.610531 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:03.612494 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:03.640722 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:03.726974 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:03.816128 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.009912 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:04.017488 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.044739 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.141494 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:04.182407 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.187363 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:04.423178 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:04.461278 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.539368 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.550062 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:04.613781 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:04.626305 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.650163 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:04.739807 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:04.787015 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.869959 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.887185 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:04.921640 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:05.002375 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:05.081996 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:05.096254 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.109757 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.116757 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.143502 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:05.265233 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:05.438222 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.520135 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.610219 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.664907 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.717083 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:05.768553 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:05.785403 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:05.808116 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:05.938989 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:06.189555 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:06.252382 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:06.254156 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.352038 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:06.426666 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.495393 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.522207 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.530977 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.543808 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:01:06.562592 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:06.638360 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.714888 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:06.858532 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:06.900964 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:01:06.920465 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:06.956228 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:06.964876 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:01:07.064145 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.079116 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:07.213233 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:07.219980 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.261536 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.286135 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:07.314702 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:07.352555 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:07.387226 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.438172 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:07.445302 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.446971 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.451340 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.537730 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:07.591470 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.760356 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:07.818032 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.902328 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:07.978502 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:08.217381 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:08.319396 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:08.401175 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:08.402407 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:08.421895 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:08.453966 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:08.529045 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:01:08.790043 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:08.794672 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:08.809080 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:08.874559 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:01:08.880334 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:08.938427 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.005908 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.032800 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.094686 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.109685 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.368294 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:09.383931 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:09.426647 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:09.444450 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.457255 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.473788 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:09.592738 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:09.645233 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:09.725535 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.765583 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.847162 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.861366 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:09.886585 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.892621 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:09.911948 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:09.943506 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.077234 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:10.103027 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.221268 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:10.228745 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:10.328972 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.443807 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:10.462816 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.625233 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.774648 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.776672 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.853948 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:10.860749 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:10.864601 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:10.922443 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:11.104584 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:11.152457 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.341475 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.372368 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.429502 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:11.499505 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.546031 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:11.625481 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:01:11.735672 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.773097 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.816959 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:11.819876 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:11.889580 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:11.908638 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:01:11.941969 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:11.987341 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.075347 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:12.135819 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.233227 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 02:01:12.262676 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.266780 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.288262 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:12.309228 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.360759 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:12.380003 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:12.402638 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.425980 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:12.462424 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.594795 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:12.662788 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:12.671289 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:12.843521 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:12.855661 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.017807 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.087890 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:13.104169 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.246287 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:13.270443 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.291589 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.346601 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.374028 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.490304 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:13.594712 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:01:13.772898 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:13.773443 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:13.786363 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:13.829784 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:13.923905 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:13.929780 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:13.943763 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.037829 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.037918 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.157642 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:14.292829 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.312839 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.324797 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:14.327337 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:14.398569 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.470844 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:14.612869 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.623471 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:14.666982 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.713048 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:14.789654 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:14.951999 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:14.961856 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:14.977973 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.022991 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:15.128086 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.209991 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:15.261246 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.316331 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:15.405668 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.496488 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.505715 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.660867 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:01:15.663986 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.716905 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:15.771301 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.773818 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:15.980692 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:15.999981 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:01:16.000464 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.097162 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:16.178997 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:16.288627 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.291013 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:16.524645 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.571014 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.599152 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.600457 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:16.713843 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:01:16.721669 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.780169 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:16.799729 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.832304 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.871310 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.889578 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:16.951370 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:16.963902 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:16.975847 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:17.032397 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:17.093663 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:17.103466 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:17.312731 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:01:17.378923 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:17.384640 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 02:01:17.478811 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:17.534174 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:17.539008 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:17.553539 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:17.591425 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:17.620250 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:17.801167 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:17.897492 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:17.910822 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:18.044996 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:18.067834 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.109045 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.131687 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:18.202491 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.261046 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.467618 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.562247 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.625131 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:01:18.665997 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:18.703735 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:18.706993 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:18.758549 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:18.912206 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:18.917043 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:01:18.918013 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.046671 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.066740 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:19.130142 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:19.171921 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.177870 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:19.206377 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.352741 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.395710 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:19.428560 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.441351 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:19.530972 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.548933 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:19.682513 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:19.754776 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:19.832952 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:20.006644 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:20.113059 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.116541 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.153497 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:20.212393 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.266512 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:20.382583 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.407833 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:20.553401 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.626842 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.654449 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.662364 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.662916 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.682853 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:20.733639 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:20.737874 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:20.866142 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:20.968168 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:21.061960 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:21.109418 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:01:21.232414 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:21.246083 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:21.337343 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:21.379938 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:21.382956 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:21.406747 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:21.496580 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:21.541633 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:21.711052 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:21.751397 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:01:21.854702 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:21.865288 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:21.882495 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.014299 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.029752 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.046702 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.106392 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:22.117091 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:22.120102 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:01:22.123543 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.341982 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:01:22.359746 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:22.400764 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.413362 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 02:01:22.455476 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:22.709681 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.712550 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:22.712731 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.786118 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.806525 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:22.942411 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:22.969322 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.982472 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:22.992177 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:23.090662 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:23.102766 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:23.217387 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:23.316299 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:23.340142 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:23.461964 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:23.501861 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:23.502689 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:23.597443 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:23.655416 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:01:23.723407 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:23.969584 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:23.992046 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:01:23.992660 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:23.993681 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.004812 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.038952 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:24.110509 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.255307 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:24.362144 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:24.378579 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:24.383697 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.452924 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:24.460189 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.489107 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.566826 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:24.629669 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.777271 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:24.818985 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:24.874507 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:24.910866 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:24.996870 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.013854 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.221911 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.331076 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:25.338335 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.405405 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.422451 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.431643 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:25.462049 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:25.476542 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:25.664075 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.673795 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.828724 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.870841 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:25.909010 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:25.939447 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:26.013809 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:26.058527 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:26.112951 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.127313 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.132268 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:01:26.291960 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.334030 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.478641 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:26.548892 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:26.625208 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.656246 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.850658 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.852782 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:26.899317 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:26.908167 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:01:26.931586 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:27.056452 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:27.120321 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.123168 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:27.154620 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.241519 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:27.245290 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:27.262761 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.280022 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:27.397621 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:27.419836 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:27.425050 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:01:27.568399 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.794456 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.807322 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.827277 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:27.856556 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.895542 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.949159 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:27.951246 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:27.958456 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:28.136791 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:28.189360 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:28.199615 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:28.376014 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:01:28.419296 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:28.451805 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:28.523510 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:28.569664 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:28.641494 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:28.686933 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:28.697491 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:28.703044 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:28.842625 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:28.899755 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:28.928139 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:29.116384 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:29.188678 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:29.247321 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:29.376878 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:29.378859 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:29.417263 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:01:29.466410 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:29.483443 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:29.542287 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:29.554821 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:29.694494 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:29.768697 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:29.780314 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:29.892045 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:29.960589 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:30.014717 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.077728 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.156351 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.180039 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.293156 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.434617 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:30.468514 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:30.474496 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:30.508226 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:30.511570 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.656613 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.678848 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.813102 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:01:30.901147 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.909157 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.956158 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:30.995314 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:31.025645 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.039152 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:31.054558 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.091018 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:31.095064 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:31.245262 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.290387 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.322366 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:31.505157 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:31.678723 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.717075 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.717780 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.842047 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.875735 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:31.973686 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:32.049140 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:01:32.084522 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:32.139699 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:32.155422 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:32.284762 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:32.344123 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:32.363644 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:32.431766 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:32.459916 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:32.459954 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:32.470670 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:01:32.555328 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:32.579814 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:32.586148 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:32.614882 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:32.738997 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:32.889951 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.023167 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:33.026037 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:33.107034 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:33.235970 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.238044 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:33.252660 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.258318 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.280176 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.311957 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.430089 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:01:33.576104 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.587960 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:33.675524 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.681411 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:33.685832 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:33.744602 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:33.836592 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:33.865147 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:33.968894 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:33.983445 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:34.034159 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:34.100183 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:34.340009 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:34.390820 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:34.438721 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:34.481469 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:34.540680 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:34.557358 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:34.579284 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:01:34.579818 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:34.632316 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:34.766543 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:34.867249 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.002388 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:35.031852 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.042459 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.122500 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:35.181260 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.317343 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.429594 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:35.498561 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:35.561573 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.596503 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.615121 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:35.633957 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.676716 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.773126 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:35.798243 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:35.866467 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:01:35.881086 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:35.912550 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:36.009150 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:36.031831 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:36.069219 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:36.158730 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:36.201370 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.283541 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.322132 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.472211 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:36.506768 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.549484 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.561738 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:36.686308 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.746755 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.954123 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.993345 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:36.998458 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:37.041329 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:37.070878 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:37.149878 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:37.210551 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:01:37.283146 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.307163 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:37.406533 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:37.410073 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:01:37.463678 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:01:37.498891 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.500241 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 02:01:37.515916 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.617777 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:37.678309 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:37.751574 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.846298 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.848783 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.874915 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:37.888293 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.010282 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:38.015650 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:38.229445 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:38.237079 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:01:38.244500 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.378893 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:38.424382 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:38.483772 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.508976 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.561885 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.658428 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.700600 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:38.777225 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:38.811089 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:38.836864 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:38.868578 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:38.897254 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:39.020826 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:39.165099 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:39.311998 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:39.429082 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:39.582177 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:39.582839 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:39.642352 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:39.647711 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:39.677656 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:39.728855 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:39.785497 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:39.843683 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:39.875890 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:39.943065 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.014163 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.021378 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:40.077111 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.121771 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.211100 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:40.252265 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.267432 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.285542 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.521783 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.522575 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:40.546074 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:40.553700 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:40.681216 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:40.749899 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:40.903671 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:01:40.925418 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:40.950623 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:40.986953 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:41.100972 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:41.208272 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:41.214474 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:41.242965 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:41.335093 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.371198 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.446585 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.474042 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.484459 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:41.512497 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:41.588707 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.762581 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.823081 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:41.937649 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:41.977106 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.038406 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:42.121588 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:42.207903 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.216665 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:01:42.377064 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.452591 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:42.457053 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.458000 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:42.479804 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:42.504276 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:42.527213 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:01:42.532661 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:42.545428 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:42.647585 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.718293 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:42.772262 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.884912 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:42.902508 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:42.938530 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.952216 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:42.988720 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:43.036325 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.249486 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:01:43.300380 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.398489 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:43.408277 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:43.422528 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.478232 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:43.488845 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:43.684393 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.719936 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.796190 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.807765 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:43.855525 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:43.941629 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:43.988963 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:43.993687 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:44.005389 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:44.214569 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:44.227063 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:44.442003 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:44.545429 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:44.557965 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:44.685667 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:44.690090 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:44.697783 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:44.732229 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:44.759894 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:44.781226 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:44.906988 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.045285 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.061677 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:45.073954 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:45.083329 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:45.146313 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:45.289733 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.312693 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.355468 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.541100 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.576630 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:45.582119 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:45.630815 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:01:45.653339 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:45.677156 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.746759 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:45.795305 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.068670 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.105912 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.131680 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:46.293503 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:46.355809 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.386225 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.445775 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:46.530299 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.538317 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:46.547363 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:46.583909 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:46.787626 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.807194 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.933390 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.933884 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:46.936115 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:46.991863 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.001099 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:47.006222 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.168197 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.177441 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.264904 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:01:47.401347 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:47.429137 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.480088 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.517308 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:47.517963 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.545374 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.548382 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:01:47.555372 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:01:47.646418 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.697980 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.711210 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.713161 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.835431 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:47.908155 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:47.952323 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:48.095335 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:48.100718 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:48.130172 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:01:48.152209 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:48.212354 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:48.275713 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:48.392874 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:48.522202 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:48.662298 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:48.699582 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:48.775103 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:48.822216 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:48.895188 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:49.070050 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:49.073041 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.083374 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:49.088389 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.148356 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:49.157298 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.391059 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.463213 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:49.498092 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:49.554574 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.556937 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:49.706984 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.732069 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:49.824213 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.883509 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.956465 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:49.974704 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:50.013751 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:01:50.014449 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:50.029904 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:50.126564 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:50.128482 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:01:50.129210 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:50.246820 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:50.268255 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:50.287715 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:50.319774 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:50.451745 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:50.513535 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:50.608453 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:50.641408 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:01:50.799615 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:50.904675 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:50.973583 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.055809 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:51.075212 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:51.256717 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.269558 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:51.292980 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.460775 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:51.504734 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.612221 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.627614 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:51.653296 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:51.688846 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.792510 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:51.830363 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:51.919072 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:51.998214 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:52.040784 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:52.044839 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.085661 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.209193 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:52.298223 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:01:52.298964 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.323904 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:01:52.324564 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.364528 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.477038 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:52.510278 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.523139 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:52.526600 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:52.527249 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.578778 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:01:52.602278 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:01:52.704478 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:52.879081 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:52.912043 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:52.944983 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:53.037214 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:53.041515 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:01:53.043462 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:53.064880 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:53.117859 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:53.283165 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:53.334118 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:53.398968 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:53.526888 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:53.652848 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:53.685477 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:53.821636 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:53.827286 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:53.902751 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:53.951654 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:53.985867 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:54.000618 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.012733 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.220916 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.243943 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:54.264755 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:54.427129 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:54.487313 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.491292 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.672178 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:54.709298 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.780139 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.793186 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.802371 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:54.834904 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:54.933735 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:54.978356 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:55.051685 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:55.117255 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:55.154117 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:55.157982 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:55.171742 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:55.315865 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:55.349396 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:55.405366 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:55.435969 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 02:01:55.445535 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:55.463038 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:55.517254 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:55.681216 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:55.870723 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:55.894635 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:55.920738 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:01:55.958750 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:55.987471 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 02:01:55.988221 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b2686f7b58341e9a0b1f2b27cfbb173: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:56.220377 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:56.310029 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:56.343461 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:56.402088 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:56.457774 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:56.516991 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:56.618841 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:56.637854 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:56.717677 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:56.846957 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:56.863612 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:56.864663 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.024497 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.025310 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.056840 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.107126 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.111343 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.172200 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.248670 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.297435 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.406004 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:01:57.430160 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:01:57.455220 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:57.465951 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.540675 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:57.541268 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.545359 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.554975 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:57.613582 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 02:01:57.621402 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:57.633159 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.844748 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.887530 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:57.892925 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:57.961961 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.022634 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:58.039899 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:58.092762 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.118610 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 781ba9bbf19e413bb8981c48073f3360: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:58.223294 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:01:58.237210 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.500681 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:58.526911 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:58.665611 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:58.665602 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.723464 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.788671 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.815141 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:58.856961 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:58.903131 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:59.058171 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.096700 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.227311 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.296926 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.338773 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.448204 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:59.466573 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:59.567958 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:59.688333 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.713711 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:59.715603 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.759555 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:01:59.850950 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:59.856377 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:01:59.881287 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:01:59.982507 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.024734 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.115087 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.239343 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:00.241796 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:00.256451 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.322631 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.412022 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:00.424121 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:00.441833 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:00.448006 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:02:00.448690 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:00.535831 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:00.636509 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.681140 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:00.721688 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.850976 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:00.862305 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:01.008031 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.140262 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:01.241991 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.273519 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:01.286617 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:01.355780 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.420258 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:01.595664 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.602509 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.720045 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.758962 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.780758 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:01.825606 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:02.074596 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.115796 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.163800 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.180651 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.275439 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:02.305730 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.339718 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.340688 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.365689 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.451861 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.498901 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.517920 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:02:02.556967 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 02:02:02.558002 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.604797 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.613813 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:02:02.711290 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:02.712601 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:02:02.713312 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:02.754818 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:02.756940 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:02.789290 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:02.874025 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:02.969530 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.143366 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:03.158504 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.230310 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.283413 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:03.345494 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.352161 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:03.566143 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.574064 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:03.606985 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:03.806108 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.807255 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:03.890158 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:03.904129 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:03.944803 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.029161 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.036108 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.064603 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.397538 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:04.401473 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.448985 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.520900 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.678184 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:04.692535 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:04.712196 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:04.823257 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:04.892858 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:04.940237 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:04.963591 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:05.079074 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.094637 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.179790 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.222776 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:05.248775 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:02:05.249436 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.278610 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:05.364894 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:05.380950 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.384580 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:05.474478 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:05.487893 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:05.628007 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:02:05.687577 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.721271 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.749960 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:05.777470 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:05.832403 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:05.941303 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:05.978210 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.033496 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:06.282347 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:06.332551 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:06.345731 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.406284 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.427008 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.659490 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:06.726891 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.839800 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.890321 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:06.895577 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:06.927152 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.051836 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.119707 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.215108 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.241345 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.372479 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.382942 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.416200 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.418255 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.446202 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:07.497484 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.549324 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:02:07.670069 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:02:07.689841 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:07.734109 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:02:07.736914 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.783263 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:07.808929 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:07.822273 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:07.852474 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:07.861039 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.874285 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.955874 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:07.963224 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:08.120363 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:08.160899 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:08.186064 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:08.330624 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:08.349201 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:08.369225 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:08.529287 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:08.604491 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:08.630620 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:08.853760 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:08.858829 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:08.867101 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:08.995957 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.141355 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.155445 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.203318 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.224418 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.420670 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:09.446147 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.453361 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.500085 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.571986 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.651212 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.752431 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.844026 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.844794 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.853516 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:09.904587 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:09.973502 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:10.006863 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.091253 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:10.212417 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:10.219067 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.358180 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:10.436529 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:10.487349 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:10.491972 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.544896 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:10.558815 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:10.684080 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.707449 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.738438 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:10.790122 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.819805 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:02:10.972793 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:10.990232 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.054720 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:11.112659 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.332533 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.336185 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.418323 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:11.584517 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:11.588445 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:11.603643 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.641969 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:11.656385 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.664595 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.775482 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:11.803870 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:12.069947 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:12.107266 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:12.107435 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:12.135335 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:12.276165 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:12.290752 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:12.319177 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:12.395121 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:12.418154 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:12.421164 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:12.516842 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:12.564675 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:02:12.644380 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:12.664543 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:12.667888 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:12.704170 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:02:12.741230 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:02:12.821442 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 02:02:12.929355 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:13.024578 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:13.034785 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:13.085500 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.118603 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:13.171629 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.186467 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:13.204926 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:02:13.234269 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.338043 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:13.339550 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.384333 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.458796 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:13.533170 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:13.751468 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.843859 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:13.877058 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:13.922174 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:14.001593 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.103229 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.115585 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:14.205538 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.255554 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.285416 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.300549 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.333088 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:14.536031 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.586299 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.589306 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:14.752117 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:14.866518 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:14.917235 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:14.931643 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:14.947578 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:14.970978 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:15.110456 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:15.171690 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:15.221014 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:15.329279 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:15.424909 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:15.453608 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:15.496876 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:15.549750 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:15.550411 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:15.584013 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:15.664975 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:15.801468 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:15.814957 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:15.873085 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:15.873803 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:15.904824 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:15.967953 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:16.027056 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:02:16.161856 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:16.259593 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:16.268016 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:16.270610 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.437029 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.559237 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.572284 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.643332 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.799249 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:16.806497 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.884970 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:16.943419 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:16.943714 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:17.008059 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:17.089216 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:17.115415 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:17.123245 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:17.132802 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:17.185685 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:17.256012 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:17.441733 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:17.530691 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:17.655951 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:17.730824 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:02:17.731541 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:17.735051 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:02:17.772605 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:17.794696 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:17.841763 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:17.848196 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:02:17.848932 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:17.858208 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:17.920543 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:18.185099 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:18.200592 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:18.202385 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.221524 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.245775 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:02:18.296494 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.335755 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:18.338145 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.361902 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.398029 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:18.636283 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.700767 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:18.723769 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:18.761963 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:18.783627 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:18.958845 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:18.984211 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:18.991828 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:19.278604 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:19.299422 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:19.436636 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:19.452555 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:19.484038 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:19.527818 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:19.600690 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:19.729702 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:19.734207 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:19.773113 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:19.877668 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:19.918994 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:19.991559 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:20.071049 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:20.172497 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:20.294926 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:20.333369 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:20.339985 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:20.362829 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:20.457074 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:20.536913 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:20.585335 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:20.625594 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:20.696004 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:20.696676 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:20.704658 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:02:20.776633 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:20.875780 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:02:20.876389 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:20.885915 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:21.033965 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:21.126557 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:21.126821 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:21.184823 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.256778 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:21.356283 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.406775 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:21.482311 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.618947 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.638612 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:21.715193 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.715273 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.814497 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:21.872699 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:21.897588 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.108668 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:22.136781 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.202836 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:22.221580 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:22.336483 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.362249 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:22.418881 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.441851 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:22.590991 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:22.593962 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:22.725244 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.743230 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:02:22.749881 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:02:22.750603 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:22.817145 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.846905 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:22.880016 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:02:22.913264 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:22.915982 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:22.925243 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:02:22.944018 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:23.045128 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:23.065001 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:02:23.232442 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:23.250523 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:23.360567 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:23.443918 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:23.476258 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:23.514748 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:23.640421 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:23.662462 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:23.715585 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:23.724076 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:23.758760 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:23.804755 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:23.942163 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.168470 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.173327 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.260062 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:24.283541 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.311331 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.383268 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.392653 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.577811 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:24.677001 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:24.719375 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.768589 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:24.777429 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.822499 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:24.871037 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:24.873703 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:24.874048 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:24.885396 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:25.121496 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:25.135321 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:25.294369 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:25.373205 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:25.440256 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:25.541972 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:25.565197 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:25.596570 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:25.679090 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:02:25.712313 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:25.830421 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:25.833472 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:25.914292 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:25.926062 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:25.993577 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:25.994294 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:26.060817 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:26.084133 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:26.188239 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.287364 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:26.393703 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.450752 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.550499 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:26.603729 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.650470 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.742869 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.859961 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.866006 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:26.878474 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:26.940127 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:27.093076 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:27.148528 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:27.169291 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:27.213975 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:27.263134 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:27.270440 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:27.284952 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:27.353014 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:27.378932 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:27.454514 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:27.532829 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:27.626361 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:27.654537 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:27.767668 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 02:02:27.789741 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:02:27.902334 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:27.947328 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:02:27.950729 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:27.968681 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:02:28.052938 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:28.123342 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:28.256246 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:28.329646 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:28.335804 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:28.391278 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:02:28.400559 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:28.453709 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:28.503863 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:28.552942 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:28.639580 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:28.730926 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:28.731529 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:28.803333 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:28.852422 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.036474 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.047356 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.059063 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.071064 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:29.216908 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.294169 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.371416 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.372896 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.430454 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.464406 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.621266 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:29.698102 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.748438 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.804065 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.819247 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.837145 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.863443 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:29.906888 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:29.985661 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:29.991815 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:30.161787 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:30.194727 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:30.433646 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:30.459946 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:30.618109 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:30.644651 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:30.691941 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:30.715719 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:30.882814 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:02:30.959712 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:31.002489 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.110961 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:02:31.111106 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:31.111683 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:31.120332 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:31.121168 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:31.138633 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:31.447834 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.456209 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:31.516099 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.518446 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.526527 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.595371 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.619977 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.668875 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:31.700032 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.750393 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:31.901471 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:32.024137 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.144136 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:32.189790 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:32.195631 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.216466 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:32.252172 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.266829 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.383632 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:32.483841 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.487372 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:32.554880 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.605798 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:32.655200 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:32.796180 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:02:32.799659 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:32.939975 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:02:32.940599 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:32.950966 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:32.952086 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:02:33.035389 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:02:33.175479 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:33.213837 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:33.257242 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:33.258209 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:33.286984 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:02:33.452646 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:33.524652 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:33.665951 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:33.743752 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:33.747419 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:33.763674 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:33.794862 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:33.858772 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:33.895344 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:33.922483 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:33.923142 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:33.999913 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.106484 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.130910 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.226804 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.257474 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.329795 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.344049 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:34.505503 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.565515 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:34.602804 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:34.639655 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:34.667481 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:34.838583 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:34.868707 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:34.921777 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:35.042790 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:35.112725 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:35.199385 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:35.229147 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:35.348807 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:35.367704 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:35.417109 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:35.663908 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:35.692756 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:35.720584 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:35.725777 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:35.740151 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:35.753916 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:35.791033 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:02:35.951719 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:36.012610 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.115166 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.273826 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:36.275442 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:36.276309 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:36.383525 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.412853 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:36.413875 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.420810 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.508559 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.648169 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.652995 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:36.683768 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:36.752856 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:36.804723 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:36.847510 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:36.864988 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:37.010985 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:37.042271 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:37.093641 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:37.194330 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:37.200286 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:37.201287 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:37.316421 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:37.378830 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:37.459479 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:37.479604 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:37.543252 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:37.741413 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:37.796342 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:37.894533 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:37.912595 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:37.991626 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:02:38.045387 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:02:38.045959 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:38.079815 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:02:38.141965 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:02:38.182119 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:38.299024 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:38.308584 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:02:38.338968 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:38.405443 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:38.511929 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:38.535712 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:38.579874 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:38.582185 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:38.736091 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:38.757990 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:38.892429 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.010128 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.022780 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:39.148501 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.284751 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.301916 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:39.320430 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:39.320911 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:39.368628 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:39.419531 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.441977 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.466647 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.588812 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:39.631613 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:39.633720 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.699997 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:39.719547 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:39.767519 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:39.933889 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:39.981060 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:40.011065 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:40.011302 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:40.237061 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:40.280282 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:40.299168 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:40.427800 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:40.572098 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:40.709734 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:40.737099 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:40.764326 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:40.929863 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:40.938045 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:40.946702 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:40.962834 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:02:40.985661 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:41.112277 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:41.140516 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:41.299932 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 02:02:41.300632 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:41.320530 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:41.329962 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:41.591781 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:41.630280 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:41.646394 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:41.724617 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:41.828262 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:41.888810 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:41.923149 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:41.966634 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:42.060745 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.104728 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.105836 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.128278 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:42.173339 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.245908 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:42.258029 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:42.291288 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:42.327180 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.375783 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.435530 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:42.479935 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:42.630292 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:42.664762 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:42.828670 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:42.922461 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:42.952469 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:43.041337 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:43.064035 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:02:43.070417 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:02:43.089120 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:43.150888 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:02:43.153668 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:43.185206 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:43.275836 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:43.452502 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:43.514149 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:43.602744 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:43.648535 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:43.663166 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:43.703012 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:43.738999 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:02:43.801604 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:43.848502 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:43.993476 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.007663 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:44.108708 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.134143 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:44.246141 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.321416 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:44.400017 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:44.451661 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:44.507822 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.510144 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.514415 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:44.540951 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:44.544032 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:44.634817 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:44.695983 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.735570 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.780090 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:44.811894 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.919046 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:44.923327 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:44.944243 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:45.026544 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:45.181783 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:45.217527 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:45.337626 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:45.427379 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:45.542145 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:45.640740 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:45.678781 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:45.763455 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:45.923698 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.106925 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:46.154357 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.182996 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.284353 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.315738 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:02:46.317049 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.353916 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.363153 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.377892 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.386336 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.404632 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.489532 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:46.490176 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:46.650257 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.695802 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.707504 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.794915 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:46.832320 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:46.872644 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:46.937055 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:46.945111 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:46.987807 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:47.070300 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:47.162719 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:47.222025 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:47.237500 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:47.330866 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:47.397548 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:47.398790 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:47.483203 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:47.655923 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:47.718722 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:47.807608 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:47.855981 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:47.885344 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:47.900624 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:47.988626 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:48.062696 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:48.091495 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:02:48.106593 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:02:48.125593 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:48.267887 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:02:48.288375 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:48.459893 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:48.526537 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:48.596634 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:02:48.607343 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:48.790498 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:48.809757 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:48.838460 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:48.862802 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:48.883576 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:48.885299 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:48.920603 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:48.955181 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:48.963595 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:48.966403 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:49.053982 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.069423 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.327888 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.335005 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:49.408731 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:49.438906 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:49.511451 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:49.655987 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.739558 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:49.784896 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:49.835312 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:49.838241 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:49.855317 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:49.875051 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.910676 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.959076 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:49.966766 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:50.117547 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:50.193445 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:50.261565 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:50.360118 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:50.430460 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:50.475466 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:50.500949 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:50.777138 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:50.838723 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:50.958159 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:51.036839 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.112284 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:02:51.139390 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.232208 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:51.302593 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.302690 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.344138 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.347543 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.391505 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.417414 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.480937 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.509276 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:51.509827 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.535971 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.684041 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.702008 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:51.725560 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:51.804284 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:51.989337 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:52.080010 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:52.113734 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.194530 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.239418 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:52.240134 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.250448 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.292321 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:52.339777 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:52.371376 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:52.514748 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.520334 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:52.579999 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.630992 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:52.796376 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:52.868448 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:52.884601 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:52.893904 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:52.904247 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:53.066423 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:53.092869 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:02:53.133565 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:53.151548 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:02:53.269752 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:53.336088 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:02:53.371971 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:53.432202 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:53.566721 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:53.612322 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:53.670279 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:02:53.794612 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:53.806284 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:53.899231 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:53.927457 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:53.936004 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:53.947839 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:53.988224 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:54.181973 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:54.182641 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:54.204053 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:54.208323 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:54.218299 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:54.369328 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:54.467178 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:54.483820 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:54.506585 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:54.545776 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:54.567592 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:54.632593 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:54.730489 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:54.749171 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:54.795951 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:55.046361 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:55.082079 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:55.101423 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:55.136162 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:55.136801 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:55.175612 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:55.206094 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:55.287020 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:55.408911 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:55.432193 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:55.452368 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:55.682977 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:55.778512 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:55.916939 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:55.996904 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 02:02:55.997538 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b2686f7b58341e9a0b1f2b27cfbb173: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:56.120044 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:56.144425 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:56.189472 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.231869 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.298738 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:56.418795 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:56.438885 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.506873 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:02:56.540742 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:56.642446 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.678660 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.688602 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.689414 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:56.690004 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:56.843644 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.858641 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:56.893101 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:56.925766 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.084055 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:57.155305 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.156404 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.156673 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:57.164316 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.196436 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.303989 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:57.351965 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.375124 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.389276 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:57.426501 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:57.492425 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.584728 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:57.693640 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:57.729216 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:57.731258 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:57.876351 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:57.909497 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:57.964098 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:58.128242 24471 catalog_manager.cc:4573] TS 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 781ba9bbf19e413bb8981c48073f3360: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:58.155988 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:58.163530 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:58.206467 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:02:58.207849 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:02:58.392881 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:58.425789 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:02:58.428645 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:58.455302 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:58.455461 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:58.675508 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:58.691108 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:58.713835 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:58.747089 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:58.801534 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:02:58.898726 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.030651 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.083048 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.175227 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.235090 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.272431 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:59.290058 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.335937 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.345144 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.402520 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.459542 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:02:59.498813 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.634131 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.704180 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.764642 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.801395 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.871505 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:02:59.873881 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:02:59.895512 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:02:59.950312 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:02:59.957957 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:00.036118 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:00.053301 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:00.104528 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:00.245744 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:03:00.316684 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:00.318763 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:00.345314 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:00.408537 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:00.721814 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:00.767163 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:00.767351 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:00.916829 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:01.133081 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.202656 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:01.207877 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.246599 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:01.297341 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:01.321280 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.329484 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:01.432806 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.607493 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.617966 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:01.636499 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.644240 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:01.698359 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:03:01.699018 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:01.814678 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:01.838832 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.898082 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:01.963039 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:02.021713 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.118410 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:02.162547 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.251711 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.262758 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.297418 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:02.303104 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:02.392071 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:02.436748 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.444422 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:02.465632 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:02.565399 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.576153 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:02.795413 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:02.851811 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:02.883998 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:02.918771 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:02.962579 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:03.198029 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:03:03.246995 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:03:03.259581 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:03.284045 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:03:03.287261 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:03.427105 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:03:03.467579 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:03.469554 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:03.479454 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:03.636679 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:03.659978 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:03.804136 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:03.850031 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:03.879336 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:03.914116 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.059780 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.081717 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.096524 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.186156 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.239590 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.279793 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.282352 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:04.349056 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.425961 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.453713 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.607151 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.612247 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.636490 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.739804 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.747860 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:04.754043 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:04.855142 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:04.884500 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.918612 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.939580 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:04.946144 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:05.087663 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:05.159529 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:05.272579 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:05.281390 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:05.288774 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:05.297767 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:05.397871 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:05.704022 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:05.796411 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:05.830533 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:05.836473 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:05.844849 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.122435 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:06.251096 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.256518 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.304786 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.348790 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.502947 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:06.506433 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.532372 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.576512 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.678476 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.708788 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 02:03:06.711819 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.726145 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.737959 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:06.898103 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.924858 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:06.949824 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.967844 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:06.978766 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:07.125605 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:07.231132 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:07.272382 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:07.332785 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:07.334156 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:07.432937 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:07.453400 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:07.465764 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:07.589277 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:07.604614 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:07.650700 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:07.651283 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:07.691159 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:07.825132 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:07.948310 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:08.085438 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:08.121486 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:08.220230 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:08.274307 27510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:03:08.274873 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:08.364440 24830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:03:08.387255 27377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:03:08.491628 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:08.616461 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:08.639219 27110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:03:08.672406 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:08.775357 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:08.777487 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:08.850536 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:08.857458 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.009896 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.019354 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:09.085177 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.119369 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:09.183176 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.254251 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.359136 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.432524 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.460961 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.462774 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.465723 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.529219 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:09.556128 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.606353 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.698877 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:09.781728 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.883441 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:09.940886 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:09.963165 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:09.992478 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:10.028527 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:10.104669 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:10.194479 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:10.313268 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:10.325140 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:10.343711 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:10.364943 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:10.505887 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:10.696337 27510 consensus_peers.cc:485] T 6e608c8945504b18b1cd82d421f2d3b9 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:10.783478 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:10.837848 24564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:03:11.028245 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:11.036013 27510 consensus_peers.cc:485] T 5b0629c2f9014b5dad64e46658818f29 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:11.043489 24830 consensus_peers.cc:485] T 92382869887b43f9a23a2c0ea0012720 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.063606 27110 consensus_peers.cc:485] T 6d2cd42bb63448238ddfcca4cce87b74 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:11.169049 24830 consensus_peers.cc:485] T b3edec06b3d8450a8cc3002be70c848a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:11.216336 27110 consensus_peers.cc:485] T 1d9ee234307a410ca578be50f798804d P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.246347 27510 consensus_peers.cc:485] T da60bdbf6831421e9f9abec53c6e1d35 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:11.297704 27377 consensus_peers.cc:485] T a85e6a0806654abd81ca5661a2d019ce P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.333854 27110 consensus_peers.cc:485] T 8919df9a8a574882bad4995fc5df7e50 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.444396 24564 consensus_peers.cc:485] T 356b84e895db449d990b1e3c45d60159 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.564654 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:11.615574 27110 consensus_peers.cc:485] T b37610aa1329476188c0cd8b92973ca6 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:11.770684 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:03:11.774304 24963 consensus_peers.cc:485] T 89580549c13d4c1ebdb2a5d24da2a01e P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.843901 27510 consensus_peers.cc:485] T 2078a814ae1648e2bde5e521ef45af6d P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.881863 24963 consensus_peers.cc:485] T a2d5afdc9ff6458ca834de042f8e14ab P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:11.905025 24830 consensus_peers.cc:485] T 16ae91433f10465c922183298a387caa P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:11.941991 27377 consensus_peers.cc:485] T 0dba43199ac941dcaefe4f0236e90156 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.063380 24830 consensus_peers.cc:485] T 7d0cc9aba3804968a6dce97682f6b38e P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:12.069828 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:12.080968 24830 consensus_peers.cc:485] T 26b7501639064035a570b3b4c1cf083f P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:03:12.132705 24830 consensus_peers.cc:485] T 48085cebf94e4f549e6174a88acdcc59 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.179963 27110 consensus_peers.cc:485] T 252502ad2e124996b345888706e674d5 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.235389 27110 consensus_peers.cc:485] T 7db1384cb0374899a6e17a550e1f7cba P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.245359 24830 consensus_peers.cc:485] T 0e47667fc8fa49e6b18ee3df06cda00a P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.368948 27377 consensus_peers.cc:485] T b6c09e8f421d4ae2adf7c9b63121943c P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:03:12.419126 24564 consensus_peers.cc:485] T 334184346780410a9f7efe298ae4d5e1 P 588bfc090a5247598cd5d3cee5ad2007 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:12.422566 24830 consensus_peers.cc:485] T 938581084b27405cbc01917e1a982208 P f46bcd2bc124471981349b1e2726253b -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.442369 27377 consensus_peers.cc:485] T 0180449195154cb49ebccb8b604af4e2 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:12.487524 24698 consensus_peers.cc:485] T ebe8db639839433d967583371351eacb P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.548221 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:12.756712 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:12.757575 24698 consensus_peers.cc:485] T 237661e1b1fa45e5a780c0af90e0138a P 92cb785c165749eca49af5784756461c -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:03:12.875591 27510 consensus_peers.cc:485] T 57e64ca6504646b8b77c5d36e32d5d6c P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:12.913834 27510 consensus_peers.cc:485] T 84ae33b0741c48c090dda6d4a819ef02 P bb653087e8554c7591cf547dd969f3be -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:03:12.965365 24963 consensus_peers.cc:462] T 9b8667a49f1a44339122040b0122be0d P e0b9e7df4e8543fa9b8e3ffc24330657 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111)
W20240502 02:03:12.995998 27110 consensus_peers.cc:485] T 056c4148f1394bfab265dd06b10d88c1 P 9dd619a595e94004ac0f7f5a9054e7cc -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:03:13.052528 27377 consensus_peers.cc:485] T d0758e7e80294f8f97342db35da3761d P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn't send request to peer 1049b4c8783b48f39be22bd113ad349f. Status: Network error: Client connection negotiation failed: client connection to 127.17.243.6:41707: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:03:13.225167 27377 consensus_peers.cc:485] T 3a5c460a54024915a2944c2f44a10b05 P 1cc0a6338e0a467dbdc6efb3e67a6a09 -> Peer 1049b4c8783b48f39be22bd113ad349f (127.17.243.6:41707): Couldn'